You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@mesos.apache.org by "Benjamin Mahler (JIRA)" <ji...@apache.org> on 2017/04/28 22:39:04 UTC
[jira] [Comment Edited] (MESOS-6345)
ExamplesTest.PersistentVolumeFramework failing due to double free
corruption on Ubuntu 14.04
[ https://issues.apache.org/jira/browse/MESOS-6345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15559169#comment-15559169 ]
Benjamin Mahler edited comment on MESOS-6345 at 4/28/17 10:38 PM:
------------------------------------------------------------------
{noformat}
[04:56:48] : [Step 10/10] [ RUN ] ExamplesTest.PersistentVolumeFramework
[04:56:48]W: [Step 10/10] I1008 04:56:48.212661 25257 master.cpp:1097] Master terminating
[04:56:48]W: [Step 10/10] I1008 04:56:48.212674 25254 status_update_manager.cpp:395] Received status update acknowledgement (UUID: 542b14f7-bfc9-4be3-81b4-c23a1da9ecb5) for task 2 of framework 84cfc7a4-ad66-4f0d-965c-33ff6093ef32-0000
[04:56:48]W: [Step 10/10] I1008 04:56:48.212709 25254 status_update_manager.cpp:531] Cleaning up status update stream for task 2 of framework 84cfc7a4-ad66-4f0d-965c-33ff6093ef32-0000
[04:56:48]W: [Step 10/10] I1008 04:56:48.212712 25257 master.cpp:7725] Removing executor 'default' with resources {} of framework 84cfc7a4-ad66-4f0d-965c-33ff6093ef32-0000 on agent 84cfc7a4-ad66-4f0d-965c-33ff6093ef32-S2 at slave(1)@172.30.2.21:52703 (ip-172-30-2-21.mesosphere.io)
[04:56:48]W: [Step 10/10] I1008 04:56:48.212767 25254 slave.cpp:2953] Status update manager successfully handled status update acknowledgement (UUID: 542b14f7-bfc9-4be3-81b4-c23a1da9ecb5) for task 2 of framework 84cfc7a4-ad66-4f0d-965c-33ff6093ef32-0000
[04:56:48]W: [Step 10/10] I1008 04:56:48.212782 25254 slave.cpp:6543] Completing task 2
[04:56:48]W: [Step 10/10] I1008 04:56:48.212792 25258 hierarchical.cpp:517] Removed agent 84cfc7a4-ad66-4f0d-965c-33ff6093ef32-S2
[04:56:48]W: [Step 10/10] I1008 04:56:48.212829 25257 master.cpp:7696] Removing task 3 with resources cpus(*):1; mem(*):128 of framework 84cfc7a4-ad66-4f0d-965c-33ff6093ef32-0000 on agent 84cfc7a4-ad66-4f0d-965c-33ff6093ef32-S1 at slave(3)@172.30.2.21:52703 (ip-172-30-2-21.mesosphere.io)
[04:56:48]W: [Step 10/10] I1008 04:56:48.212888 25257 master.cpp:7725] Removing executor 'default' with resources {} of framework 84cfc7a4-ad66-4f0d-965c-33ff6093ef32-0000 on agent 84cfc7a4-ad66-4f0d-965c-33ff6093ef32-S1 at slave(3)@172.30.2.21:52703 (ip-172-30-2-21.mesosphere.io)
[04:56:48]W: [Step 10/10] I1008 04:56:48.212915 25258 hierarchical.cpp:517] Removed agent 84cfc7a4-ad66-4f0d-965c-33ff6093ef32-S1
[04:56:48]W: [Step 10/10] I1008 04:56:48.213017 25257 master.cpp:7725] Removing executor 'default' with resources {} of framework 84cfc7a4-ad66-4f0d-965c-33ff6093ef32-0000 on agent 84cfc7a4-ad66-4f0d-965c-33ff6093ef32-S0 at slave(2)@172.30.2.21:52703 (ip-172-30-2-21.mesosphere.io)
[04:56:48]W: [Step 10/10] I1008 04:56:48.213102 25254 hierarchical.cpp:517] Removed agent 84cfc7a4-ad66-4f0d-965c-33ff6093ef32-S0
[04:56:48]W: [Step 10/10] I1008 04:56:48.213281 25251 hierarchical.cpp:337] Removed framework 84cfc7a4-ad66-4f0d-965c-33ff6093ef32-0000
[04:56:48]W: [Step 10/10] I1008 04:56:48.213404 25257 slave.cpp:4174] Got exited event for master@172.30.2.21:52703
[04:56:48]W: [Step 10/10] I1008 04:56:48.213418 25253 slave.cpp:4174] Got exited event for master@172.30.2.21:52703
[04:56:48]W: [Step 10/10] W1008 04:56:48.213426 25257 slave.cpp:4179] Master disconnected! Waiting for a new master to be elected
[04:56:48]W: [Step 10/10] W1008 04:56:48.213433 25253 slave.cpp:4179] Master disconnected! Waiting for a new master to be elected
[04:56:48]W: [Step 10/10] I1008 04:56:48.213407 25254 slave.cpp:4174] Got exited event for master@172.30.2.21:52703
[04:56:48]W: [Step 10/10] W1008 04:56:48.213448 25254 slave.cpp:4179] Master disconnected! Waiting for a new master to be elected
[04:56:48]W: [Step 10/10] I1008 04:56:48.214047 25254 slave.cpp:787] Agent terminating
[04:56:48]W: [Step 10/10] I1008 04:56:48.214068 25254 slave.cpp:2506] Asked to shut down framework 84cfc7a4-ad66-4f0d-965c-33ff6093ef32-0000 by @0.0.0.0:0
[04:56:48]W: [Step 10/10] I1008 04:56:48.214076 25254 slave.cpp:2531] Shutting down framework 84cfc7a4-ad66-4f0d-965c-33ff6093ef32-0000
[04:56:48]W: [Step 10/10] I1008 04:56:48.214083 25254 slave.cpp:4855] Shutting down executor 'default' of framework 84cfc7a4-ad66-4f0d-965c-33ff6093ef32-0000 (via HTTP)
[04:56:48]W: [Step 10/10] E1008 04:56:48.215160 25384 executor.cpp:681] End-Of-File received from agent. The agent closed the event stream
[04:56:48]W: [Step 10/10] I1008 04:56:48.215250 25254 slave.cpp:787] Agent terminating
[04:56:48]W: [Step 10/10] I1008 04:56:48.215266 25254 slave.cpp:2506] Asked to shut down framework 84cfc7a4-ad66-4f0d-965c-33ff6093ef32-0000 by @0.0.0.0:0
[04:56:48]W: [Step 10/10] I1008 04:56:48.215279 25254 slave.cpp:2531] Shutting down framework 84cfc7a4-ad66-4f0d-965c-33ff6093ef32-0000
[04:56:48]W: [Step 10/10] I1008 04:56:48.215291 25254 slave.cpp:4855] Shutting down executor 'default' of framework 84cfc7a4-ad66-4f0d-965c-33ff6093ef32-0000 (via HTTP)
[04:56:48]W: [Step 10/10] E1008 04:56:48.216333 25366 executor.cpp:681] End-Of-File received from agent. The agent closed the event stream
[04:56:48]W: [Step 10/10] I1008 04:56:48.216490 25226 slave.cpp:787] Agent terminating
[04:56:48]W: [Step 10/10] I1008 04:56:48.216507 25226 slave.cpp:2506] Asked to shut down framework 84cfc7a4-ad66-4f0d-965c-33ff6093ef32-0000 by @0.0.0.0:0
[04:56:48]W: [Step 10/10] I1008 04:56:48.216518 25226 slave.cpp:2531] Shutting down framework 84cfc7a4-ad66-4f0d-965c-33ff6093ef32-0000
[04:56:48]W: [Step 10/10] I1008 04:56:48.216524 25226 slave.cpp:4855] Shutting down executor 'default' of framework 84cfc7a4-ad66-4f0d-965c-33ff6093ef32-0000 (via HTTP)
[04:56:48]W: [Step 10/10] E1008 04:56:48.217491 25387 executor.cpp:681] End-Of-File received from agent. The agent closed the event stream
[04:56:48]W: [Step 10/10] Using temporary directory '/mnt/teamcity/temp/buildTmp/ExamplesTest_PersistentVolumeFramework_zqfSTm'
[04:56:48]W: [Step 10/10] /mnt/teamcity/work/4240ba9ddd0997c3/src/tests/persistent_volume_framework_test.sh: line 20: /mnt/teamcity/work/4240ba9ddd0997c3/build/src/colors.sh: No such file or directory
[04:56:48]W: [Step 10/10] /mnt/teamcity/work/4240ba9ddd0997c3/src/tests/persistent_volume_framework_test.sh: line 21: /mnt/teamcity/work/4240ba9ddd0997c3/build/src/atexit.sh: No such file or directory
[04:56:48]W: [Step 10/10] I1008 04:56:48.558832 25396 logging.cpp:199] Logging to STDERR
[04:56:48]W: [Step 10/10] I1008 04:56:48.561102 25396 process.cpp:1069] libprocess is initialized on 172.30.2.21:45167 with 8 worker threads
[04:56:48]W: [Step 10/10] I1008 04:56:48.575021 25396 leveldb.cpp:174] Opened db in 12.462333ms
[04:56:48]W: [Step 10/10] I1008 04:56:48.582849 25396 leveldb.cpp:181] Compacted db in 7.809821ms
[04:56:48]W: [Step 10/10] I1008 04:56:48.582872 25396 leveldb.cpp:196] Created db iterator in 8910ns
[04:56:48]W: [Step 10/10] I1008 04:56:48.582878 25396 leveldb.cpp:202] Seeked to beginning of db in 807ns
[04:56:48]W: [Step 10/10] I1008 04:56:48.582882 25396 leveldb.cpp:271] Iterated through 0 keys in the db in 418ns
[04:56:48]W: [Step 10/10] I1008 04:56:48.582906 25396 replica.cpp:776] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned
[04:56:48]W: [Step 10/10] I1008 04:56:48.583354 25423 recover.cpp:451] Starting replica recovery
[04:56:48]W: [Step 10/10] I1008 04:56:48.583447 25425 recover.cpp:477] Replica is in EMPTY status
[04:56:48]W: [Step 10/10] I1008 04:56:48.583479 25396 local.cpp:255] Creating default 'local' authorizer
[04:56:48]W: [Step 10/10] I1008 04:56:48.583894 25421 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from __req_res__(1)@172.30.2.21:45167
[04:56:48]W: [Step 10/10] I1008 04:56:48.584003 25421 recover.cpp:197] Received a recover response from a replica in EMPTY status
[04:56:48]W: [Step 10/10] I1008 04:56:48.584157 25426 recover.cpp:568] Updating replica status to STARTING
[04:56:48]W: [Step 10/10] I1008 04:56:48.584314 25421 master.cpp:380] Master c5e5ed69-9ab4-460d-99c8-d613dca680a9 (ip-172-30-2-21.mesosphere.io) started on 172.30.2.21:45167
[04:56:48]W: [Step 10/10] I1008 04:56:48.584326 25421 master.cpp:382] Flags at startup: --acls="permissive: true
[04:56:48]W: [Step 10/10] register_frameworks {
[04:56:48]W: [Step 10/10] principals {
[04:56:48]W: [Step 10/10] type: ANY
[04:56:48]W: [Step 10/10] }
[04:56:48]W: [Step 10/10] roles {
[04:56:48]W: [Step 10/10] type: SOME
[04:56:48]W: [Step 10/10] values: "test"
[04:56:48]W: [Step 10/10] }
[04:56:48]W: [Step 10/10] }
[04:56:48]W: [Step 10/10] " --agent_ping_timeout="15secs" --agent_reregister_timeout="10mins" --allocation_interval="1secs" --allocator="HierarchicalDRF" --authenticate_agents="false" --authenticate_frameworks="false" --authenticate_http_frameworks="false" --authenticate_http_readonly="false" --authenticate_http_readwrite="false" --authenticators="crammd5" --authorizers="local" --credentials="/mnt/teamcity/temp/buildTmp/ExamplesTest_PersistentVolumeFramework_zqfSTm/credentials" --framework_sorter="drf" --help="true" --hostname_lookup="true" --http_authenticators="basic" --initialize_driver_logging="true" --log_auto_initialize="true" --logbufsecs="0" --logging_level="INFO" --max_agent_ping_timeouts="5" --max_completed_frameworks="50" --max_completed_tasks_per_framework="1000" --quiet="false" --recovery_agent_removal_limit="100%" --registry="replicated_log" --registry_fetch_timeout="1mins" --registry_gc_interval="15mins" --registry_max_agent_age="2weeks" --registry_max_agent_count="102400" --registry_store_timeout="20secs" --registry_strict="false" --roles="test" --root_submissions="true" --user_sorter="drf" --version="false" --webui_dir="/mnt/teamcity/work/4240ba9ddd0997c3/src/webui" --work_dir="/mnt/teamcity/temp/buildTmp/mesos-8KiPML" --zk_session_timeout="10secs"
[04:56:48]W: [Step 10/10] I1008 04:56:48.584619 25421 master.cpp:434] Master allowing unauthenticated frameworks to register
[04:56:48]W: [Step 10/10] I1008 04:56:48.584630 25421 master.cpp:448] Master allowing unauthenticated agents to register
[04:56:48]W: [Step 10/10] I1008 04:56:48.584636 25421 master.cpp:462] Master allowing HTTP frameworks to register without authentication
[04:56:48]W: [Step 10/10] I1008 04:56:48.584645 25421 credentials.hpp:37] Loading credentials for authentication from '/mnt/teamcity/temp/buildTmp/ExamplesTest_PersistentVolumeFramework_zqfSTm/credentials'
[04:56:48]W: [Step 10/10] I1008 04:56:48.584748 25421 master.cpp:504] Using default 'crammd5' authenticator
[04:56:48]W: [Step 10/10] I1008 04:56:48.584777 25421 authenticator.cpp:519] Initializing server SASL
[04:56:48]W: [Step 10/10] I1008 04:56:48.585579 25396 containerizer.cpp:202] Using isolation: filesystem/posix,posix/cpu,posix/mem,network/cni
[04:56:48]W: [Step 10/10] I1008 04:56:48.587158 25426 slave.cpp:208] Mesos agent started on (1)@172.30.2.21:45167
[04:56:48]W: [Step 10/10] I1008 04:56:48.587236 25421 auxprop.cpp:73] Initialized in-memory auxiliary property plugin
[04:56:48]W: [Step 10/10] I1008 04:56:48.587273 25421 master.cpp:584] Authorization enabled
[04:56:48]W: [Step 10/10] W1008 04:56:48.587286 25421 master.cpp:647] The '--roles' flag is deprecated. This flag will be removed in the future. See the Mesos 0.27 upgrade notes for more information
[04:56:48]W: [Step 10/10] I1008 04:56:48.587389 25422 hierarchical.cpp:149] Initialized hierarchical allocator process
[04:56:48]W: [Step 10/10] I1008 04:56:48.587393 25427 whitelist_watcher.cpp:77] No whitelist given
[04:56:48]W: [Step 10/10] I1008 04:56:48.587170 25426 slave.cpp:209] Flags at startup: --acls="permissive: true
[04:56:48]W: [Step 10/10] register_frameworks {
[04:56:48]W: [Step 10/10] principals {
[04:56:48]W: [Step 10/10] type: ANY
[04:56:48]W: [Step 10/10] }
[04:56:48]W: [Step 10/10] roles {
[04:56:48]W: [Step 10/10] type: SOME
[04:56:48]W: [Step 10/10] values: "test"
[04:56:48]W: [Step 10/10] }
[04:56:48]W: [Step 10/10] }
[04:56:48]W: [Step 10/10] " --appc_simple_discovery_uri_prefix="http://" --appc_store_dir="/tmp/mesos/store/appc" --authenticate_http_readonly="false" --authenticate_http_readwrite="false" --authenticatee="crammd5" --authentication_backoff_factor="1secs" --authorizer="local" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="mesos" --default_role="test" --disk_watch_interval="1mins" --docker="docker" --docker_kill_orphans="true" --docker_registry="https://registry-1.docker.io" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker" --docker_volume_checkpoint_dir="/var/run/mesos/isolators/docker/volume" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/mesos/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --hostname_lookup="true" --http_authenticators="basic" --http_command_executor="false" --image_provisioner_backend="copy" --initialize_driver_logging="true" --isolation="filesystem/posix,posix/cpu,posix/mem" --launcher="posix" --launcher_dir="/mnt/teamcity/work/4240ba9ddd0997c3/build/src" --logbufsecs="0" --logging_level="INFO" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="1secs" --revocable_cpu_low_priority="true" --runtime_dir="/mnt/teamcity/temp/buildTmp/mesos-gkgAKP" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_enable_support="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/mnt/teamcity/temp/buildTmp/mesos-8KiPML/0"
[04:56:48]W: [Step 10/10] I1008 04:56:48.587808 25426 slave.cpp:533] Agent resources: cpus(test):8; mem(test):14015; disk(test):75375; ports(test):[31000-32000]
[04:56:48]W: [Step 10/10] I1008 04:56:48.587857 25426 slave.cpp:541] Agent attributes: [ ]
[04:56:48]W: [Step 10/10] I1008 04:56:48.587870 25426 slave.cpp:546] Agent hostname: ip-172-30-2-21.mesosphere.io
[04:56:48]W: [Step 10/10] I1008 04:56:48.588235 25396 containerizer.cpp:202] Using isolation: filesystem/posix,posix/cpu,posix/mem,network/cni
[04:56:48]W: [Step 10/10] I1008 04:56:48.588538 25428 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 4.284018ms
[04:56:48]W: [Step 10/10] I1008 04:56:48.588562 25428 replica.cpp:320] Persisted replica status to STARTING
[04:56:48]W: [Step 10/10] I1008 04:56:48.588670 25425 recover.cpp:477] Replica is in STARTING status
[04:56:48]W: [Step 10/10] I1008 04:56:48.588866 25424 replica.cpp:673] Replica in STARTING status received a broadcasted recover request from __req_res__(2)@172.30.2.21:45167
[04:56:48]W: [Step 10/10] I1008 04:56:48.588955 25424 recover.cpp:197] Received a recover response from a replica in STARTING status
[04:56:48]W: [Step 10/10] I1008 04:56:48.589053 25428 recover.cpp:568] Updating replica status to VOTING
[04:56:48]W: [Step 10/10] I1008 04:56:48.589443 25423 slave.cpp:208] Mesos agent started on (2)@172.30.2.21:45167
[04:56:48]W: [Step 10/10] I1008 04:56:48.589452 25423 slave.cpp:209] Flags at startup: --acls="permissive: true
[04:56:48]W: [Step 10/10] register_frameworks {
[04:56:48]W: [Step 10/10] principals {
[04:56:48]W: [Step 10/10] type: ANY
[04:56:48]W: [Step 10/10] }
[04:56:48]W: [Step 10/10] roles {
[04:56:48]W: [Step 10/10] type: SOME
[04:56:48]W: [Step 10/10] values: "test"
[04:56:48]W: [Step 10/10] }
[04:56:48]W: [Step 10/10] }
[04:56:48]W: [Step 10/10] " --appc_simple_discovery_uri_prefix="http://" --appc_store_dir="/tmp/mesos/store/appc" --authenticate_http_readonly="false" --authenticate_http_readwrite="false" --authenticatee="crammd5" --authentication_backoff_factor="1secs" --authorizer="local" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="mesos" --default_role="test" --disk_watch_interval="1mins" --docker="docker" --docker_kill_orphans="true" --docker_registry="https://registry-1.docker.io" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker" --docker_volume_checkpoint_dir="/var/run/mesos/isolators/docker/volume" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/mesos/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --hostname_lookup="true" --http_authenticators="basic" --http_command_executor="false" --image_provisioner_backend="copy" --initialize_driver_logging="true" --isolation="filesystem/posix,posix/cpu,posix/mem" --launcher="posix" --launcher_dir="/mnt/teamcity/work/4240ba9ddd0997c3/build/src" --logbufsecs="0" --logging_level="INFO" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="1secs" --revocable_cpu_low_priority="true" --runtime_dir="/mnt/teamcity/temp/buildTmp/mesos-gkgAKP" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_enable_support="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/mnt/teamcity/temp/buildTmp/mesos-8KiPML/1"
[04:56:48]W: [Step 10/10] I1008 04:56:48.589845 25423 slave.cpp:533] Agent resources: cpus(test):8; mem(test):14015; disk(test):75375; ports(test):[31000-32000]
[04:56:48]W: [Step 10/10] I1008 04:56:48.589870 25423 slave.cpp:541] Agent attributes: [ ]
[04:56:48]W: [Step 10/10] I1008 04:56:48.589874 25423 slave.cpp:546] Agent hostname: ip-172-30-2-21.mesosphere.io
[04:56:48]W: [Step 10/10] I1008 04:56:48.590322 25396 containerizer.cpp:202] Using isolation: filesystem/posix,posix/cpu,posix/mem,network/cni
[04:56:48]W: [Step 10/10] I1008 04:56:48.591367 25424 slave.cpp:208] Mesos agent started on (3)@172.30.2.21:45167
[04:56:48]W: [Step 10/10] I1008 04:56:48.591378 25424 slave.cpp:209] Flags at startup: --acls="permissive: true
[04:56:48]W: [Step 10/10] register_frameworks {
[04:56:48]W: [Step 10/10] principals {
[04:56:48]W: [Step 10/10] type: ANY
[04:56:48]W: [Step 10/10] }
[04:56:48]W: [Step 10/10] roles {
[04:56:48]W: [Step 10/10] type: SOME
[04:56:48]W: [Step 10/10] values: "test"
[04:56:48]W: [Step 10/10] }
[04:56:48]W: [Step 10/10] }
[04:56:48]W: [Step 10/10] " --appc_simple_discovery_uri_prefix="http://" --appc_store_dir="/tmp/mesos/store/appc" --authenticate_http_readonly="false" --authenticate_http_readwrite="false" --authenticatee="crammd5" --authentication_backoff_factor="1secs" --authorizer="local" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="mesos" --default_role="test" --disk_watch_interval="1mins" --docker="docker" --docker_kill_orphans="true" --docker_registry="https://registry-1.docker.io" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker" --docker_volume_checkpoint_dir="/var/run/mesos/isolators/docker/volume" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/mesos/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="true" --hostname_lookup="true" --http_authenticators="basic" --http_command_executor="false" --image_provisioner_backend="copy" --initialize_driver_logging="true" --isolation="filesystem/posix,posix/cpu,posix/mem" --launcher="posix" --launcher_dir="/mnt/teamcity/work/4240ba9ddd0997c3/build/src" --logbufsecs="0" --logging_level="INFO" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="1secs" --revocable_cpu_low_priority="true" --runtime_dir="/mnt/teamcity/temp/buildTmp/mesos-gkgAKP" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_enable_support="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/mnt/teamcity/temp/buildTmp/mesos-8KiPML/2"
[04:56:48]W: [Step 10/10] I1008 04:56:48.591719 25424 slave.cpp:533] Agent resources: cpus(test):8; mem(test):14015; disk(test):75375; ports(test):[31000-32000]
[04:56:48]W: [Step 10/10] I1008 04:56:48.591742 25424 slave.cpp:541] Agent attributes: [ ]
[04:56:48]W: [Step 10/10] I1008 04:56:48.591744 25424 slave.cpp:546] Agent hostname: ip-172-30-2-21.mesosphere.io
[04:56:48]W: [Step 10/10] *** Error in `/mnt/teamcity/work/4240ba9ddd0997c3/build/src/.libs/lt-persistent-volume-framework': double free or corruption (fasttop): 0x00007f1ae0006a20 ***
[04:56:48]W: [Step 10/10] *** Aborted at 1475902608 (unix time) try "date -d @1475902608" if you are using GNU date ***
[04:56:48]W: [Step 10/10] I1008 04:56:48.592744 25425 state.cpp:57] Recovering state from '/mnt/teamcity/temp/buildTmp/mesos-8KiPML/2/meta'
[04:56:48]W: [Step 10/10] I1008 04:56:48.592808 25423 state.cpp:57] Recovering state from '/mnt/teamcity/temp/buildTmp/mesos-8KiPML/1/meta'
[04:56:48]W: [Step 10/10] I1008 04:56:48.592952 25425 status_update_manager.cpp:203] Recovering status update manager
[04:56:48]W: [Step 10/10] I1008 04:56:48.592957 25423 status_update_manager.cpp:203] Recovering status update manager
[04:56:48]W: [Step 10/10] I1008 04:56:48.593010 25424 containerizer.cpp:557] Recovering containerizer
[04:56:48]W: [Step 10/10] I1008 04:56:48.593143 25396 sched.cpp:226] Version: 1.1.0
[04:56:48]W: [Step 10/10] I1008 04:56:48.593158 25425 master.cpp:2013] Elected as the leading master!
[04:56:48]W: [Step 10/10] I1008 04:56:48.593173 25425 master.cpp:1560] Recovering from registrar
[04:56:48]W: [Step 10/10] I1008 04:56:48.593211 25424 registrar.cpp:329] Recovering registrar
[04:56:48]W: [Step 10/10] I1008 04:56:48.593250 25425 sched.cpp:330] New master detected at master@172.30.2.21:45167
[04:56:48]W: [Step 10/10] I1008 04:56:48.593282 25425 sched.cpp:341] No credentials provided. Attempting to register without authentication
[04:56:48]W: [Step 10/10] I1008 04:56:48.593293 25425 sched.cpp:820] Sending SUBSCRIBE call to master@172.30.2.21:45167
[04:56:48]W: [Step 10/10] PC: @ 0x7f1b0bbaccc9 (unknown)
[04:56:48]W: [Step 10/10] I1008 04:56:48.593339 25425 sched.cpp:853] Will retry registration in 32.354951ms if necessary
[04:56:48]W: [Step 10/10] I1008 04:56:48.593364 25421 master.cpp:1387] Dropping 'mesos.scheduler.Call' message since not recovered yet
[04:56:48]W: [Step 10/10] I1008 04:56:48.593413 25428 provisioner.cpp:253] Provisioner recovery complete
[04:56:48]W: [Step 10/10] *** SIGABRT (@0x6334) received by PID 25396 (TID 0x7f1b02ed6700) from PID 25396; stack trace: ***
[04:56:48]W: [Step 10/10] I1008 04:56:48.593520 25421 containerizer.cpp:557] Recovering containerizer
[04:56:48]W: [Step 10/10] I1008 04:56:48.593529 25425 slave.cpp:5276] Finished recovery
[04:56:48]W: [Step 10/10] I1008 04:56:48.593627 25422 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 4.546422ms
[04:56:48]W: [Step 10/10] I1008 04:56:48.593695 25428 provisioner.cpp:253] Provisioner recovery complete
[04:56:48]W: [Step 10/10] I1008 04:56:48.593701 25422 replica.cpp:320] Persisted replica status to VOTING
[04:56:48]W: [Step 10/10] I1008 04:56:48.593760 25424 slave.cpp:5276] Finished recovery
[04:56:48]W: [Step 10/10] I1008 04:56:48.593864 25427 recover.cpp:582] Successfully joined the Paxos group
[04:56:48]W: [Step 10/10] I1008 04:56:48.593896 25425 slave.cpp:5448] Querying resource estimator for oversubscribable resources
[04:56:48]W: [Step 10/10] I1008 04:56:48.593922 25427 recover.cpp:466] Recover process terminated
[04:56:48]W: [Step 10/10] I1008 04:56:48.593976 25427 slave.cpp:5462] Received oversubscribable resources {} from the resource estimator
[04:56:48]W: [Step 10/10] I1008 04:56:48.594002 25424 slave.cpp:5448] Querying resource estimator for oversubscribable resources
[04:56:48]W: [Step 10/10] I1008 04:56:48.594017 25422 log.cpp:553] Attempting to start the writer
[04:56:48]W: [Step 10/10] I1008 04:56:48.594030 25428 status_update_manager.cpp:177] Pausing sending status updates
[04:56:48]W: [Step 10/10] I1008 04:56:48.594032 25427 slave.cpp:915] New master detected at master@172.30.2.21:45167
[04:56:48]W: [Step 10/10] I1008 04:56:48.594055 25423 slave.cpp:915] New master detected at master@172.30.2.21:45167
[04:56:48]W: [Step 10/10] I1008 04:56:48.594048 25428 status_update_manager.cpp:177] Pausing sending status updates
[04:56:48]W: [Step 10/10] I1008 04:56:48.594061 25427 slave.cpp:936] No credentials provided. Attempting to register without authentication
[04:56:48]W: [Step 10/10] I1008 04:56:48.594106 25427 slave.cpp:947] Detecting new master
[04:56:48]W: [Step 10/10] I1008 04:56:48.594071 25423 slave.cpp:936] No credentials provided. Attempting to register without authentication
[04:56:48]W: [Step 10/10] @ 0x7f1b0bf4b340 (unknown)
[04:56:48]W: [Step 10/10] I1008 04:56:48.594125 25423 slave.cpp:947] Detecting new master
[04:56:48]W: [Step 10/10] I1008 04:56:48.594194 25423 slave.cpp:5462] Received oversubscribable resources {} from the resource estimator
[04:56:48]W: [Step 10/10] I1008 04:56:48.594378 25422 replica.cpp:493] Replica received implicit promise request from __req_res__(3)@172.30.2.21:45167 with proposal 1
[04:56:48]W: [Step 10/10] @ 0x7f1b0bbaccc9 (unknown)
[04:56:48]W: [Step 10/10] @ 0x7f1b0bbb00d8 (unknown)
[04:56:48]W: [Step 10/10] @ 0x7f1b0bbe9394 (unknown)
[04:56:48]W: [Step 10/10] I1008 04:56:48.595368 25422 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 972334ns
[04:56:48]W: [Step 10/10] I1008 04:56:48.595381 25422 replica.cpp:342] Persisted promised to 1
[04:56:48]W: [Step 10/10] @ 0x7f1b0bbf566e (unknown)
[04:56:48]W: [Step 10/10] @ 0x7f1b0d930925 _ZNSt14_Function_base13_Base_managerIZN7process5deferIN5mesos8internal5slave5SlaveEiiSt12_PlaceholderILi1EES7_ILi2EEEENS1_9_DeferredIDTcl4bindadsrSt8functionIFvT0_T1_EEclcvSF__Efp1_fp2_EEEERKNS1_3PIDIT_EEMSJ_FvSC_SD_ET2_T3_EUliiE_E10_M_managerERSt9_Any_dataRKST_St18_Manager_operation
[04:56:48]W: [Step 10/10] I1008 04:56:48.597909 25421 coordinator.cpp:238] Coordinator attempting to fill missing positions
[04:56:48]W: [Step 10/10] I1008 04:56:48.598273 25423 replica.cpp:388] Replica received explicit promise request from __req_res__(4)@172.30.2.21:45167 for position 0 with proposal 2
[04:56:48]W: [Step 10/10] @ 0x7f1b0d935b0b std::_Function_base::_Base_manager<>::_M_manager()
[04:56:48]W: [Step 10/10] @ 0x7f1b0d8f2516 os::internal::configureSignal()
[04:56:48]W: [Step 10/10] I1008 04:56:48.599318 25423 leveldb.cpp:341] Persisting action (8 bytes) to leveldb took 1.024957ms
[04:56:48]W: [Step 10/10] I1008 04:56:48.599333 25423 replica.cpp:708] Persisted action NOP at position 0
[04:56:48]W: [Step 10/10] I1008 04:56:48.599630 25428 replica.cpp:537] Replica received write request for position 0 from __req_res__(5)@172.30.2.21:45167
[04:56:48]W: [Step 10/10] I1008 04:56:48.599660 25428 leveldb.cpp:436] Reading position from leveldb took 16893ns
[04:56:48]W: [Step 10/10] @ 0x7f1b0d904022 mesos::internal::slave::Slave::initialize()
[04:56:48]W: [Step 10/10] @ 0x7f1b0e0c6ed1 process::ProcessManager::resume()
[04:56:48]W: [Step 10/10] @ 0x7f1b0e0c7187 _ZNSt6thread5_ImplISt12_Bind_simpleIFZN7process14ProcessManager12init_threadsEvEUt_vEEE6_M_runEv
[04:56:48]W: [Step 10/10] @ 0x7f1b0c726a60 (unknown)
[04:56:48]W: [Step 10/10] @ 0x7f1b0bf43182 start_thread
[04:56:48]W: [Step 10/10] @ 0x7f1b0bc7047d (unknown)
[04:56:49] : [Step 10/10] ../../src/tests/script.cpp:80: Failure
[04:56:49] : [Step 10/10] Failed
[04:56:49] : [Step 10/10] persistent_volume_framework_test.sh terminated with signal Aborted
{noformat}
was (Author: avinash@mesosphere.io):
```
[04:56:48] : [Step 10/10] [ RUN ] ExamplesTest.PersistentVolumeFramework
[04:56:48]W: [Step 10/10] I1008 04:56:48.212661 25257 master.cpp:1097] Master terminating
[04:56:48]W: [Step 10/10] I1008 04:56:48.212674 25254 status_update_manager.cpp:395] Received status update acknowledgement (UUID: 542b14f7-bfc9-4be3-81b4-c23a1da9ecb5) for task 2 of framework 84cfc7a4-ad66-4f0d-965c-33ff6093ef32-0000
[04:56:48]W: [Step 10/10] I1008 04:56:48.212709 25254 status_update_manager.cpp:531] Cleaning up status update stream for task 2 of framework 84cfc7a4-ad66-4f0d-965c-33ff6093ef32-0000
[04:56:48]W: [Step 10/10] I1008 04:56:48.212712 25257 master.cpp:7725] Removing executor 'default' with resources {} of framework 84cfc7a4-ad66-4f0d-965c-33ff6093ef32-0000 on agent 84cfc7a4-ad66-4f0d-965c-33ff6093ef32-S2 at slave(1)@172.30.2.21:52703 (ip-172-30-2-21.mesosphere.io)
[04:56:48]W: [Step 10/10] I1008 04:56:48.212767 25254 slave.cpp:2953] Status update manager successfully handled status update acknowledgement (UUID: 542b14f7-bfc9-4be3-81b4-c23a1da9ecb5) for task 2 of framework 84cfc7a4-ad66-4f0d-965c-33ff6093ef32-0000
[04:56:48]W: [Step 10/10] I1008 04:56:48.212782 25254 slave.cpp:6543] Completing task 2
[04:56:48]W: [Step 10/10] I1008 04:56:48.212792 25258 hierarchical.cpp:517] Removed agent 84cfc7a4-ad66-4f0d-965c-33ff6093ef32-S2
[04:56:48]W: [Step 10/10] I1008 04:56:48.212829 25257 master.cpp:7696] Removing task 3 with resources cpus(*):1; mem(*):128 of framework 84cfc7a4-ad66-4f0d-965c-33ff6093ef32-0000 on agent 84cfc7a4-ad66-4f0d-965c-33ff6093ef32-S1 at slave(3)@172.30.2.21:52703 (ip-172-30-2-21.mesosphere.io)
[04:56:48]W: [Step 10/10] I1008 04:56:48.212888 25257 master.cpp:7725] Removing executor 'default' with resources {} of framework 84cfc7a4-ad66-4f0d-965c-33ff6093ef32-0000 on agent 84cfc7a4-ad66-4f0d-965c-33ff6093ef32-S1 at slave(3)@172.30.2.21:52703 (ip-172-30-2-21.mesosphere.io)
[04:56:48]W: [Step 10/10] I1008 04:56:48.212915 25258 hierarchical.cpp:517] Removed agent 84cfc7a4-ad66-4f0d-965c-33ff6093ef32-S1
[04:56:48]W: [Step 10/10] I1008 04:56:48.213017 25257 master.cpp:7725] Removing executor 'default' with resources {} of framework 84cfc7a4-ad66-4f0d-965c-33ff6093ef32-0000 on agent 84cfc7a4-ad66-4f0d-965c-33ff6093ef32-S0 at slave(2)@172.30.2.21:52703 (ip-172-30-2-21.mesosphere.io)
[04:56:48]W: [Step 10/10] I1008 04:56:48.213102 25254 hierarchical.cpp:517] Removed agent 84cfc7a4-ad66-4f0d-965c-33ff6093ef32-S0
[04:56:48]W: [Step 10/10] I1008 04:56:48.213281 25251 hierarchical.cpp:337] Removed framework 84cfc7a4-ad66-4f0d-965c-33ff6093ef32-0000
[04:56:48]W: [Step 10/10] I1008 04:56:48.213404 25257 slave.cpp:4174] Got exited event for master@172.30.2.21:52703
[04:56:48]W: [Step 10/10] I1008 04:56:48.213418 25253 slave.cpp:4174] Got exited event for master@172.30.2.21:52703
[04:56:48]W: [Step 10/10] W1008 04:56:48.213426 25257 slave.cpp:4179] Master disconnected! Waiting for a new master to be elected
[04:56:48]W: [Step 10/10] W1008 04:56:48.213433 25253 slave.cpp:4179] Master disconnected! Waiting for a new master to be elected
[04:56:48]W: [Step 10/10] I1008 04:56:48.213407 25254 slave.cpp:4174] Got exited event for master@172.30.2.21:52703
[04:56:48]W: [Step 10/10] W1008 04:56:48.213448 25254 slave.cpp:4179] Master disconnected! Waiting for a new master to be elected
[04:56:48]W: [Step 10/10] I1008 04:56:48.214047 25254 slave.cpp:787] Agent terminating
[04:56:48]W: [Step 10/10] I1008 04:56:48.214068 25254 slave.cpp:2506] Asked to shut down framework 84cfc7a4-ad66-4f0d-965c-33ff6093ef32-0000 by @0.0.0.0:0
[04:56:48]W: [Step 10/10] I1008 04:56:48.214076 25254 slave.cpp:2531] Shutting down framework 84cfc7a4-ad66-4f0d-965c-33ff6093ef32-0000
[04:56:48]W: [Step 10/10] I1008 04:56:48.214083 25254 slave.cpp:4855] Shutting down executor 'default' of framework 84cfc7a4-ad66-4f0d-965c-33ff6093ef32-0000 (via HTTP)
[04:56:48]W: [Step 10/10] E1008 04:56:48.215160 25384 executor.cpp:681] End-Of-File received from agent. The agent closed the event stream
[04:56:48]W: [Step 10/10] I1008 04:56:48.215250 25254 slave.cpp:787] Agent terminating
[04:56:48]W: [Step 10/10] I1008 04:56:48.215266 25254 slave.cpp:2506] Asked to shut down framework 84cfc7a4-ad66-4f0d-965c-33ff6093ef32-0000 by @0.0.0.0:0
[04:56:48]W: [Step 10/10] I1008 04:56:48.215279 25254 slave.cpp:2531] Shutting down framework 84cfc7a4-ad66-4f0d-965c-33ff6093ef32-0000
[04:56:48]W: [Step 10/10] I1008 04:56:48.215291 25254 slave.cpp:4855] Shutting down executor 'default' of framework 84cfc7a4-ad66-4f0d-965c-33ff6093ef32-0000 (via HTTP)
[04:56:48]W: [Step 10/10] E1008 04:56:48.216333 25366 executor.cpp:681] End-Of-File received from agent. The agent closed the event stream
[04:56:48]W: [Step 10/10] I1008 04:56:48.216490 25226 slave.cpp:787] Agent terminating
[04:56:48]W: [Step 10/10] I1008 04:56:48.216507 25226 slave.cpp:2506] Asked to shut down framework 84cfc7a4-ad66-4f0d-965c-33ff6093ef32-0000 by @0.0.0.0:0
[04:56:48]W: [Step 10/10] I1008 04:56:48.216518 25226 slave.cpp:2531] Shutting down framework 84cfc7a4-ad66-4f0d-965c-33ff6093ef32-0000
[04:56:48]W: [Step 10/10] I1008 04:56:48.216524 25226 slave.cpp:4855] Shutting down executor 'default' of framework 84cfc7a4-ad66-4f0d-965c-33ff6093ef32-0000 (via HTTP)
[04:56:48]W: [Step 10/10] E1008 04:56:48.217491 25387 executor.cpp:681] End-Of-File received from agent. The agent closed the event stream
[04:56:48]W: [Step 10/10] Using temporary directory '/mnt/teamcity/temp/buildTmp/ExamplesTest_PersistentVolumeFramework_zqfSTm'
[04:56:48]W: [Step 10/10] /mnt/teamcity/work/4240ba9ddd0997c3/src/tests/persistent_volume_framework_test.sh: line 20: /mnt/teamcity/work/4240ba9ddd0997c3/build/src/colors.sh: No such file or directory
[04:56:48]W: [Step 10/10] /mnt/teamcity/work/4240ba9ddd0997c3/src/tests/persistent_volume_framework_test.sh: line 21: /mnt/teamcity/work/4240ba9ddd0997c3/build/src/atexit.sh: No such file or directory
[04:56:48]W: [Step 10/10] I1008 04:56:48.558832 25396 logging.cpp:199] Logging to STDERR
[04:56:48]W: [Step 10/10] I1008 04:56:48.561102 25396 process.cpp:1069] libprocess is initialized on 172.30.2.21:45167 with 8 worker threads
[04:56:48]W: [Step 10/10] I1008 04:56:48.575021 25396 leveldb.cpp:174] Opened db in 12.462333ms
[04:56:48]W: [Step 10/10] I1008 04:56:48.582849 25396 leveldb.cpp:181] Compacted db in 7.809821ms
[04:56:48]W: [Step 10/10] I1008 04:56:48.582872 25396 leveldb.cpp:196] Created db iterator in 8910ns
[04:56:48]W: [Step 10/10] I1008 04:56:48.582878 25396 leveldb.cpp:202] Seeked to beginning of db in 807ns
[04:56:48]W: [Step 10/10] I1008 04:56:48.582882 25396 leveldb.cpp:271] Iterated through 0 keys in the db in 418ns
[04:56:48]W: [Step 10/10] I1008 04:56:48.582906 25396 replica.cpp:776] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned
[04:56:48]W: [Step 10/10] I1008 04:56:48.583354 25423 recover.cpp:451] Starting replica recovery
[04:56:48]W: [Step 10/10] I1008 04:56:48.583447 25425 recover.cpp:477] Replica is in EMPTY status
[04:56:48]W: [Step 10/10] I1008 04:56:48.583479 25396 local.cpp:255] Creating default 'local' authorizer
[04:56:48]W: [Step 10/10] I1008 04:56:48.583894 25421 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from __req_res__(1)@172.30.2.21:45167
[04:56:48]W: [Step 10/10] I1008 04:56:48.584003 25421 recover.cpp:197] Received a recover response from a replica in EMPTY status
[04:56:48]W: [Step 10/10] I1008 04:56:48.584157 25426 recover.cpp:568] Updating replica status to STARTING
[04:56:48]W: [Step 10/10] I1008 04:56:48.584314 25421 master.cpp:380] Master c5e5ed69-9ab4-460d-99c8-d613dca680a9 (ip-172-30-2-21.mesosphere.io) started on 172.30.2.21:45167
[04:56:48]W: [Step 10/10] I1008 04:56:48.584326 25421 master.cpp:382] Flags at startup: --acls="permissive: true
[04:56:48]W: [Step 10/10] register_frameworks {
[04:56:48]W: [Step 10/10] principals {
[04:56:48]W: [Step 10/10] type: ANY
[04:56:48]W: [Step 10/10] }
[04:56:48]W: [Step 10/10] roles {
[04:56:48]W: [Step 10/10] type: SOME
[04:56:48]W: [Step 10/10] values: "test"
[04:56:48]W: [Step 10/10] }
[04:56:48]W: [Step 10/10] }
[04:56:48]W: [Step 10/10] " --agent_ping_timeout="15secs" --agent_reregister_timeout="10mins" --allocation_interval="1secs" --allocator="HierarchicalDRF" --authenticate_agents="false" --authenticate_frameworks="false" --authenticate_http_frameworks="false" --authenticate_http_readonly="false" --authenticate_http_readwrite="false" --authenticators="crammd5" --authorizers="local" --credentials="/mnt/teamcity/temp/buildTmp/ExamplesTest_PersistentVolumeFramework_zqfSTm/credentials" --framework_sorter="drf" --help="true" --hostname_lookup="true" --http_authenticators="basic" --initialize_driver_logging="true" --log_auto_initialize="true" --logbufsecs="0" --logging_level="INFO" --max_agent_ping_timeouts="5" --max_completed_frameworks="50" --max_completed_tasks_per_framework="1000" --quiet="false" --recovery_agent_removal_limit="100%" --registry="replicated_log" --registry_fetch_timeout="1mins" --registry_gc_interval="15mins" --registry_max_agent_age="2weeks" --registry_max_agent_count="102400" --registry_store_timeout="20secs" --registry_strict="false" --roles="test" --root_submissions="true" --user_sorter="drf" --version="false" --webui_dir="/mnt/teamcity/work/4240ba9ddd0997c3/src/webui" --work_dir="/mnt/teamcity/temp/buildTmp/mesos-8KiPML" --zk_session_timeout="10secs"
[04:56:48]W: [Step 10/10] I1008 04:56:48.584619 25421 master.cpp:434] Master allowing unauthenticated frameworks to register
[04:56:48]W: [Step 10/10] I1008 04:56:48.584630 25421 master.cpp:448] Master allowing unauthenticated agents to register
[04:56:48]W: [Step 10/10] I1008 04:56:48.584636 25421 master.cpp:462] Master allowing HTTP frameworks to register without authentication
[04:56:48]W: [Step 10/10] I1008 04:56:48.584645 25421 credentials.hpp:37] Loading credentials for authentication from '/mnt/teamcity/temp/buildTmp/ExamplesTest_PersistentVolumeFramework_zqfSTm/credentials'
[04:56:48]W: [Step 10/10] I1008 04:56:48.584748 25421 master.cpp:504] Using default 'crammd5' authenticator
[04:56:48]W: [Step 10/10] I1008 04:56:48.584777 25421 authenticator.cpp:519] Initializing server SASL
[04:56:48]W: [Step 10/10] I1008 04:56:48.585579 25396 containerizer.cpp:202] Using isolation: filesystem/posix,posix/cpu,posix/mem,network/cni
[04:56:48]W: [Step 10/10] I1008 04:56:48.587158 25426 slave.cpp:208] Mesos agent started on (1)@172.30.2.21:45167
[04:56:48]W: [Step 10/10] I1008 04:56:48.587236 25421 auxprop.cpp:73] Initialized in-memory auxiliary property plugin
[04:56:48]W: [Step 10/10] I1008 04:56:48.587273 25421 master.cpp:584] Authorization enabled
[04:56:48]W: [Step 10/10] W1008 04:56:48.587286 25421 master.cpp:647] The '--roles' flag is deprecated. This flag will be removed in the future. See the Mesos 0.27 upgrade notes for more information
[04:56:48]W: [Step 10/10] I1008 04:56:48.587389 25422 hierarchical.cpp:149] Initialized hierarchical allocator process
[04:56:48]W: [Step 10/10] I1008 04:56:48.587393 25427 whitelist_watcher.cpp:77] No whitelist given
[04:56:48]W: [Step 10/10] I1008 04:56:48.587170 25426 slave.cpp:209] Flags at startup: --acls="permissive: true
[04:56:48]W: [Step 10/10] register_frameworks {
[04:56:48]W: [Step 10/10] principals {
[04:56:48]W: [Step 10/10] type: ANY
[04:56:48]W: [Step 10/10] }
[04:56:48]W: [Step 10/10] roles {
[04:56:48]W: [Step 10/10] type: SOME
[04:56:48]W: [Step 10/10] values: "test"
[04:56:48]W: [Step 10/10] }
[04:56:48]W: [Step 10/10] }
[04:56:48]W: [Step 10/10] " --appc_simple_discovery_uri_prefix="http://" --appc_store_dir="/tmp/mesos/store/appc" --authenticate_http_readonly="false" --authenticate_http_readwrite="false" --authenticatee="crammd5" --authentication_backoff_factor="1secs" --authorizer="local" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="mesos" --default_role="test" --disk_watch_interval="1mins" --docker="docker" --docker_kill_orphans="true" --docker_registry="https://registry-1.docker.io" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker" --docker_volume_checkpoint_dir="/var/run/mesos/isolators/docker/volume" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/mesos/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --hostname_lookup="true" --http_authenticators="basic" --http_command_executor="false" --image_provisioner_backend="copy" --initialize_driver_logging="true" --isolation="filesystem/posix,posix/cpu,posix/mem" --launcher="posix" --launcher_dir="/mnt/teamcity/work/4240ba9ddd0997c3/build/src" --logbufsecs="0" --logging_level="INFO" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="1secs" --revocable_cpu_low_priority="true" --runtime_dir="/mnt/teamcity/temp/buildTmp/mesos-gkgAKP" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_enable_support="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/mnt/teamcity/temp/buildTmp/mesos-8KiPML/0"
[04:56:48]W: [Step 10/10] I1008 04:56:48.587808 25426 slave.cpp:533] Agent resources: cpus(test):8; mem(test):14015; disk(test):75375; ports(test):[31000-32000]
[04:56:48]W: [Step 10/10] I1008 04:56:48.587857 25426 slave.cpp:541] Agent attributes: [ ]
[04:56:48]W: [Step 10/10] I1008 04:56:48.587870 25426 slave.cpp:546] Agent hostname: ip-172-30-2-21.mesosphere.io
[04:56:48]W: [Step 10/10] I1008 04:56:48.588235 25396 containerizer.cpp:202] Using isolation: filesystem/posix,posix/cpu,posix/mem,network/cni
[04:56:48]W: [Step 10/10] I1008 04:56:48.588538 25428 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 4.284018ms
[04:56:48]W: [Step 10/10] I1008 04:56:48.588562 25428 replica.cpp:320] Persisted replica status to STARTING
[04:56:48]W: [Step 10/10] I1008 04:56:48.588670 25425 recover.cpp:477] Replica is in STARTING status
[04:56:48]W: [Step 10/10] I1008 04:56:48.588866 25424 replica.cpp:673] Replica in STARTING status received a broadcasted recover request from __req_res__(2)@172.30.2.21:45167
[04:56:48]W: [Step 10/10] I1008 04:56:48.588955 25424 recover.cpp:197] Received a recover response from a replica in STARTING status
[04:56:48]W: [Step 10/10] I1008 04:56:48.589053 25428 recover.cpp:568] Updating replica status to VOTING
[04:56:48]W: [Step 10/10] I1008 04:56:48.589443 25423 slave.cpp:208] Mesos agent started on (2)@172.30.2.21:45167
[04:56:48]W: [Step 10/10] I1008 04:56:48.589452 25423 slave.cpp:209] Flags at startup: --acls="permissive: true
[04:56:48]W: [Step 10/10] register_frameworks {
[04:56:48]W: [Step 10/10] principals {
[04:56:48]W: [Step 10/10] type: ANY
[04:56:48]W: [Step 10/10] }
[04:56:48]W: [Step 10/10] roles {
[04:56:48]W: [Step 10/10] type: SOME
[04:56:48]W: [Step 10/10] values: "test"
[04:56:48]W: [Step 10/10] }
[04:56:48]W: [Step 10/10] }
[04:56:48]W: [Step 10/10] " --appc_simple_discovery_uri_prefix="http://" --appc_store_dir="/tmp/mesos/store/appc" --authenticate_http_readonly="false" --authenticate_http_readwrite="false" --authenticatee="crammd5" --authentication_backoff_factor="1secs" --authorizer="local" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="mesos" --default_role="test" --disk_watch_interval="1mins" --docker="docker" --docker_kill_orphans="true" --docker_registry="https://registry-1.docker.io" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker" --docker_volume_checkpoint_dir="/var/run/mesos/isolators/docker/volume" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/mesos/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --hostname_lookup="true" --http_authenticators="basic" --http_command_executor="false" --image_provisioner_backend="copy" --initialize_driver_logging="true" --isolation="filesystem/posix,posix/cpu,posix/mem" --launcher="posix" --launcher_dir="/mnt/teamcity/work/4240ba9ddd0997c3/build/src" --logbufsecs="0" --logging_level="INFO" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="1secs" --revocable_cpu_low_priority="true" --runtime_dir="/mnt/teamcity/temp/buildTmp/mesos-gkgAKP" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_enable_support="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/mnt/teamcity/temp/buildTmp/mesos-8KiPML/1"
[04:56:48]W: [Step 10/10] I1008 04:56:48.589845 25423 slave.cpp:533] Agent resources: cpus(test):8; mem(test):14015; disk(test):75375; ports(test):[31000-32000]
[04:56:48]W: [Step 10/10] I1008 04:56:48.589870 25423 slave.cpp:541] Agent attributes: [ ]
[04:56:48]W: [Step 10/10] I1008 04:56:48.589874 25423 slave.cpp:546] Agent hostname: ip-172-30-2-21.mesosphere.io
[04:56:48]W: [Step 10/10] I1008 04:56:48.590322 25396 containerizer.cpp:202] Using isolation: filesystem/posix,posix/cpu,posix/mem,network/cni
[04:56:48]W: [Step 10/10] I1008 04:56:48.591367 25424 slave.cpp:208] Mesos agent started on (3)@172.30.2.21:45167
[04:56:48]W: [Step 10/10] I1008 04:56:48.591378 25424 slave.cpp:209] Flags at startup: --acls="permissive: true
[04:56:48]W: [Step 10/10] register_frameworks {
[04:56:48]W: [Step 10/10] principals {
[04:56:48]W: [Step 10/10] type: ANY
[04:56:48]W: [Step 10/10] }
[04:56:48]W: [Step 10/10] roles {
[04:56:48]W: [Step 10/10] type: SOME
[04:56:48]W: [Step 10/10] values: "test"
[04:56:48]W: [Step 10/10] }
[04:56:48]W: [Step 10/10] }
[04:56:48]W: [Step 10/10] " --appc_simple_discovery_uri_prefix="http://" --appc_store_dir="/tmp/mesos/store/appc" --authenticate_http_readonly="false" --authenticate_http_readwrite="false" --authenticatee="crammd5" --authentication_backoff_factor="1secs" --authorizer="local" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="mesos" --default_role="test" --disk_watch_interval="1mins" --docker="docker" --docker_kill_orphans="true" --docker_registry="https://registry-1.docker.io" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker" --docker_volume_checkpoint_dir="/var/run/mesos/isolators/docker/volume" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/mesos/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="true" --hostname_lookup="true" --http_authenticators="basic" --http_command_executor="false" --image_provisioner_backend="copy" --initialize_driver_logging="true" --isolation="filesystem/posix,posix/cpu,posix/mem" --launcher="posix" --launcher_dir="/mnt/teamcity/work/4240ba9ddd0997c3/build/src" --logbufsecs="0" --logging_level="INFO" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="1secs" --revocable_cpu_low_priority="true" --runtime_dir="/mnt/teamcity/temp/buildTmp/mesos-gkgAKP" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_enable_support="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/mnt/teamcity/temp/buildTmp/mesos-8KiPML/2"
[04:56:48]W: [Step 10/10] I1008 04:56:48.591719 25424 slave.cpp:533] Agent resources: cpus(test):8; mem(test):14015; disk(test):75375; ports(test):[31000-32000]
[04:56:48]W: [Step 10/10] I1008 04:56:48.591742 25424 slave.cpp:541] Agent attributes: [ ]
[04:56:48]W: [Step 10/10] I1008 04:56:48.591744 25424 slave.cpp:546] Agent hostname: ip-172-30-2-21.mesosphere.io
[04:56:48]W: [Step 10/10] *** Error in `/mnt/teamcity/work/4240ba9ddd0997c3/build/src/.libs/lt-persistent-volume-framework': double free or corruption (fasttop): 0x00007f1ae0006a20 ***
[04:56:48]W: [Step 10/10] *** Aborted at 1475902608 (unix time) try "date -d @1475902608" if you are using GNU date ***
[04:56:48]W: [Step 10/10] I1008 04:56:48.592744 25425 state.cpp:57] Recovering state from '/mnt/teamcity/temp/buildTmp/mesos-8KiPML/2/meta'
[04:56:48]W: [Step 10/10] I1008 04:56:48.592808 25423 state.cpp:57] Recovering state from '/mnt/teamcity/temp/buildTmp/mesos-8KiPML/1/meta'
[04:56:48]W: [Step 10/10] I1008 04:56:48.592952 25425 status_update_manager.cpp:203] Recovering status update manager
[04:56:48]W: [Step 10/10] I1008 04:56:48.592957 25423 status_update_manager.cpp:203] Recovering status update manager
[04:56:48]W: [Step 10/10] I1008 04:56:48.593010 25424 containerizer.cpp:557] Recovering containerizer
[04:56:48]W: [Step 10/10] I1008 04:56:48.593143 25396 sched.cpp:226] Version: 1.1.0
[04:56:48]W: [Step 10/10] I1008 04:56:48.593158 25425 master.cpp:2013] Elected as the leading master!
[04:56:48]W: [Step 10/10] I1008 04:56:48.593173 25425 master.cpp:1560] Recovering from registrar
[04:56:48]W: [Step 10/10] I1008 04:56:48.593211 25424 registrar.cpp:329] Recovering registrar
[04:56:48]W: [Step 10/10] I1008 04:56:48.593250 25425 sched.cpp:330] New master detected at master@172.30.2.21:45167
[04:56:48]W: [Step 10/10] I1008 04:56:48.593282 25425 sched.cpp:341] No credentials provided. Attempting to register without authentication
[04:56:48]W: [Step 10/10] I1008 04:56:48.593293 25425 sched.cpp:820] Sending SUBSCRIBE call to master@172.30.2.21:45167
[04:56:48]W: [Step 10/10] PC: @ 0x7f1b0bbaccc9 (unknown)
[04:56:48]W: [Step 10/10] I1008 04:56:48.593339 25425 sched.cpp:853] Will retry registration in 32.354951ms if necessary
[04:56:48]W: [Step 10/10] I1008 04:56:48.593364 25421 master.cpp:1387] Dropping 'mesos.scheduler.Call' message since not recovered yet
[04:56:48]W: [Step 10/10] I1008 04:56:48.593413 25428 provisioner.cpp:253] Provisioner recovery complete
[04:56:48]W: [Step 10/10] *** SIGABRT (@0x6334) received by PID 25396 (TID 0x7f1b02ed6700) from PID 25396; stack trace: ***
[04:56:48]W: [Step 10/10] I1008 04:56:48.593520 25421 containerizer.cpp:557] Recovering containerizer
[04:56:48]W: [Step 10/10] I1008 04:56:48.593529 25425 slave.cpp:5276] Finished recovery
[04:56:48]W: [Step 10/10] I1008 04:56:48.593627 25422 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 4.546422ms
[04:56:48]W: [Step 10/10] I1008 04:56:48.593695 25428 provisioner.cpp:253] Provisioner recovery complete
[04:56:48]W: [Step 10/10] I1008 04:56:48.593701 25422 replica.cpp:320] Persisted replica status to VOTING
[04:56:48]W: [Step 10/10] I1008 04:56:48.593760 25424 slave.cpp:5276] Finished recovery
[04:56:48]W: [Step 10/10] I1008 04:56:48.593864 25427 recover.cpp:582] Successfully joined the Paxos group
[04:56:48]W: [Step 10/10] I1008 04:56:48.593896 25425 slave.cpp:5448] Querying resource estimator for oversubscribable resources
[04:56:48]W: [Step 10/10] I1008 04:56:48.593922 25427 recover.cpp:466] Recover process terminated
[04:56:48]W: [Step 10/10] I1008 04:56:48.593976 25427 slave.cpp:5462] Received oversubscribable resources {} from the resource estimator
[04:56:48]W: [Step 10/10] I1008 04:56:48.594002 25424 slave.cpp:5448] Querying resource estimator for oversubscribable resources
[04:56:48]W: [Step 10/10] I1008 04:56:48.594017 25422 log.cpp:553] Attempting to start the writer
[04:56:48]W: [Step 10/10] I1008 04:56:48.594030 25428 status_update_manager.cpp:177] Pausing sending status updates
[04:56:48]W: [Step 10/10] I1008 04:56:48.594032 25427 slave.cpp:915] New master detected at master@172.30.2.21:45167
[04:56:48]W: [Step 10/10] I1008 04:56:48.594055 25423 slave.cpp:915] New master detected at master@172.30.2.21:45167
[04:56:48]W: [Step 10/10] I1008 04:56:48.594048 25428 status_update_manager.cpp:177] Pausing sending status updates
[04:56:48]W: [Step 10/10] I1008 04:56:48.594061 25427 slave.cpp:936] No credentials provided. Attempting to register without authentication
[04:56:48]W: [Step 10/10] I1008 04:56:48.594106 25427 slave.cpp:947] Detecting new master
[04:56:48]W: [Step 10/10] I1008 04:56:48.594071 25423 slave.cpp:936] No credentials provided. Attempting to register without authentication
[04:56:48]W: [Step 10/10] @ 0x7f1b0bf4b340 (unknown)
[04:56:48]W: [Step 10/10] I1008 04:56:48.594125 25423 slave.cpp:947] Detecting new master
[04:56:48]W: [Step 10/10] I1008 04:56:48.594194 25423 slave.cpp:5462] Received oversubscribable resources {} from the resource estimator
[04:56:48]W: [Step 10/10] I1008 04:56:48.594378 25422 replica.cpp:493] Replica received implicit promise request from __req_res__(3)@172.30.2.21:45167 with proposal 1
[04:56:48]W: [Step 10/10] @ 0x7f1b0bbaccc9 (unknown)
[04:56:48]W: [Step 10/10] @ 0x7f1b0bbb00d8 (unknown)
[04:56:48]W: [Step 10/10] @ 0x7f1b0bbe9394 (unknown)
[04:56:48]W: [Step 10/10] I1008 04:56:48.595368 25422 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 972334ns
[04:56:48]W: [Step 10/10] I1008 04:56:48.595381 25422 replica.cpp:342] Persisted promised to 1
[04:56:48]W: [Step 10/10] @ 0x7f1b0bbf566e (unknown)
[04:56:48]W: [Step 10/10] @ 0x7f1b0d930925 _ZNSt14_Function_base13_Base_managerIZN7process5deferIN5mesos8internal5slave5SlaveEiiSt12_PlaceholderILi1EES7_ILi2EEEENS1_9_DeferredIDTcl4bindadsrSt8functionIFvT0_T1_EEclcvSF__Efp1_fp2_EEEERKNS1_3PIDIT_EEMSJ_FvSC_SD_ET2_T3_EUliiE_E10_M_managerERSt9_Any_dataRKST_St18_Manager_operation
[04:56:48]W: [Step 10/10] I1008 04:56:48.597909 25421 coordinator.cpp:238] Coordinator attempting to fill missing positions
[04:56:48]W: [Step 10/10] I1008 04:56:48.598273 25423 replica.cpp:388] Replica received explicit promise request from __req_res__(4)@172.30.2.21:45167 for position 0 with proposal 2
[04:56:48]W: [Step 10/10] @ 0x7f1b0d935b0b std::_Function_base::_Base_manager<>::_M_manager()
[04:56:48]W: [Step 10/10] @ 0x7f1b0d8f2516 os::internal::configureSignal()
[04:56:48]W: [Step 10/10] I1008 04:56:48.599318 25423 leveldb.cpp:341] Persisting action (8 bytes) to leveldb took 1.024957ms
[04:56:48]W: [Step 10/10] I1008 04:56:48.599333 25423 replica.cpp:708] Persisted action NOP at position 0
[04:56:48]W: [Step 10/10] I1008 04:56:48.599630 25428 replica.cpp:537] Replica received write request for position 0 from __req_res__(5)@172.30.2.21:45167
[04:56:48]W: [Step 10/10] I1008 04:56:48.599660 25428 leveldb.cpp:436] Reading position from leveldb took 16893ns
[04:56:48]W: [Step 10/10] @ 0x7f1b0d904022 mesos::internal::slave::Slave::initialize()
[04:56:48]W: [Step 10/10] @ 0x7f1b0e0c6ed1 process::ProcessManager::resume()
[04:56:48]W: [Step 10/10] @ 0x7f1b0e0c7187 _ZNSt6thread5_ImplISt12_Bind_simpleIFZN7process14ProcessManager12init_threadsEvEUt_vEEE6_M_runEv
[04:56:48]W: [Step 10/10] @ 0x7f1b0c726a60 (unknown)
[04:56:48]W: [Step 10/10] @ 0x7f1b0bf43182 start_thread
[04:56:48]W: [Step 10/10] @ 0x7f1b0bc7047d (unknown)
[04:56:49] : [Step 10/10] ../../src/tests/script.cpp:80: Failure
[04:56:49] : [Step 10/10] Failed
[04:56:49] : [Step 10/10] persistent_volume_framework_test.sh terminated with signal Aborted
```
> ExamplesTest.PersistentVolumeFramework failing due to double free corruption on Ubuntu 14.04
> --------------------------------------------------------------------------------------------
>
> Key: MESOS-6345
> URL: https://issues.apache.org/jira/browse/MESOS-6345
> Project: Mesos
> Issue Type: Bug
> Components: framework
> Reporter: Avinash Sridharan
> Labels: mesosphere
>
> PersistentVolumeFramework tests if failing on Ubuntu 14
> {code}
> [Step 10/10] *** Error in `/mnt/teamcity/work/4240ba9ddd0997c3/build/src/.libs/lt-persistent-volume-framework': double free or corruption (fasttop): 0x00007f1ae0006a20 ***
> [04:56:48]W: [Step 10/10] *** Aborted at 1475902608 (unix time) try "date -d @1475902608" if you are using GNU date ***
> [04:56:48]W: [Step 10/10] I1008 04:56:48.592744 25425 state.cpp:57] Recovering state from '/mnt/teamcity/temp/buildTmp/mesos-8KiPML/2/meta'
> [04:56:48]W: [Step 10/10] I1008 04:56:48.592808 25423 state.cpp:57] Recovering state from '/mnt/teamcity/temp/buildTmp/mesos-8KiPML/1/meta'
> [04:56:48]W: [Step 10/10] I1008 04:56:48.592952 25425 status_update_manager.cpp:203] Recovering status update manager
> [04:56:48]W: [Step 10/10] I1008 04:56:48.592957 25423 status_update_manager.cpp:203] Recovering status update manager
> [04:56:48]W: [Step 10/10] I1008 04:56:48.593010 25424 containerizer.cpp:557] Recovering containerizer
> [04:56:48]W: [Step 10/10] I1008 04:56:48.593143 25396 sched.cpp:226] Version: 1.1.0
> [04:56:48]W: [Step 10/10] I1008 04:56:48.593158 25425 master.cpp:2013] Elected as the leading master!
> [04:56:48]W: [Step 10/10] I1008 04:56:48.593173 25425 master.cpp:1560] Recovering from registrar
> [04:56:48]W: [Step 10/10] I1008 04:56:48.593211 25424 registrar.cpp:329] Recovering registrar
> [04:56:48]W: [Step 10/10] I1008 04:56:48.593250 25425 sched.cpp:330] New master detected at master@172.30.2.21:45167
> [04:56:48]W: [Step 10/10] I1008 04:56:48.593282 25425 sched.cpp:341] No credentials provided. Attempting to register without authentication
> [04:56:48]W: [Step 10/10] I1008 04:56:48.593293 25425 sched.cpp:820] Sending SUBSCRIBE call to master@172.30.2.21:45167
> [04:56:48]W: [Step 10/10] PC: @ 0x7f1b0bbaccc9 (unknown)
> [04:56:48]W: [Step 10/10] I1008 04:56:48.593339 25425 sched.cpp:853] Will retry registration in 32.354951ms if necessary
> [04:56:48]W: [Step 10/10] I1008 04:56:48.593364 25421 master.cpp:1387] Dropping 'mesos.scheduler.Call' message since not recovered yet
> [04:56:48]W: [Step 10/10] I1008 04:56:48.593413 25428 provisioner.cpp:253] Provisioner recovery complete
> [04:56:48]W: [Step 10/10] *** SIGABRT (@0x6334) received by PID 25396 (TID 0x7f1b02ed6700) from PID 25396; stack trace: ***
> [04:56:48]W: [Step 10/10] I1008 04:56:48.593520 25421 containerizer.cpp:557] Recovering containerizer
> [04:56:48]W: [Step 10/10] I1008 04:56:48.593529 25425 slave.cpp:5276] Finished recovery
> [04:56:48]W: [Step 10/10] I1008 04:56:48.593627 25422 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 4.546422ms
> [04:56:48]W: [Step 10/10] I1008 04:56:48.593695 25428 provisioner.cpp:253] Provisioner recovery complete
> [04:56:48]W: [Step 10/10] I1008 04:56:48.593701 25422 replica.cpp:320] Persisted replica status to VOTING
> [04:56:48]W: [Step 10/10] I1008 04:56:48.593760 25424 slave.cpp:5276] Finished recovery
> [04:56:48]W: [Step 10/10] I1008 04:56:48.593864 25427 recover.cpp:582] Successfully joined the Paxos group
> [04:56:48]W: [Step 10/10] I1008 04:56:48.593896 25425 slave.cpp:5448] Querying resource estimator for oversubscribable resources
> [04:56:48]W: [Step 10/10] I1008 04:56:48.593922 25427 recover.cpp:466] Recover process terminated
> [04:56:48]W: [Step 10/10] I1008 04:56:48.593976 25427 slave.cpp:5462] Received oversubscribable resources {} from the resource estimator
> [04:56:48]W: [Step 10/10] I1008 04:56:48.594002 25424 slave.cpp:5448] Querying resource estimator for oversubscribable resources
> [04:56:48]W: [Step 10/10] I1008 04:56:48.594017 25422 log.cpp:553] Attempting to start the writer
> [04:56:48]W: [Step 10/10] I1008 04:56:48.594030 25428 status_update_manager.cpp:177] Pausing sending status updates
> [04:56:48]W: [Step 10/10] I1008 04:56:48.594032 25427 slave.cpp:915] New master detected at master@172.30.2.21:45167
> [04:56:48]W: [Step 10/10] I1008 04:56:48.594055 25423 slave.cpp:915] New master detected at master@172.30.2.21:45167
> [04:56:48]W: [Step 10/10] I1008 04:56:48.594048 25428 status_update_manager.cpp:177] Pausing sending status updates
> [04:56:48]W: [Step 10/10] I1008 04:56:48.594061 25427 slave.cpp:936] No credentials provided. Attempting to register without authentication
> [04:56:48]W: [Step 10/10] I1008 04:56:48.594106 25427 slave.cpp:947] Detecting new master
> [04:56:48]W: [Step 10/10] I1008 04:56:48.594071 25423 slave.cpp:936] No credentials provided. Attempting to register without authentication
> [04:56:48]W: [Step 10/10] @ 0x7f1b0bf4b340 (unknown)
> [04:56:48]W: [Step 10/10] I1008 04:56:48.594125 25423 slave.cpp:947] Detecting new master
> [04:56:48]W: [Step 10/10] I1008 04:56:48.594194 25423 slave.cpp:5462] Received oversubscribable resources {} from the resource estimator
> [04:56:48]W: [Step 10/10] I1008 04:56:48.594378 25422 replica.cpp:493] Replica received implicit promise request from __req_res__(3)@172.30.2.21:45167 with proposal 1
> [04:56:48]W: [Step 10/10] @ 0x7f1b0bbaccc9 (unknown)
> [04:56:48]W: [Step 10/10] @ 0x7f1b0bbb00d8 (unknown)
> [04:56:48]W: [Step 10/10] @ 0x7f1b0bbe9394 (unknown)
> [04:56:48]W: [Step 10/10] I1008 04:56:48.595368 25422 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 972334ns
> [04:56:48]W: [Step 10/10] I1008 04:56:48.595381 25422 replica.cpp:342] Persisted promised to 1
> [04:56:48]W: [Step 10/10] @ 0x7f1b0bbf566e (unknown)
> [04:56:48]W: [Step 10/10] @ 0x7f1b0d930925 _ZNSt14_Function_base13_Base_managerIZN7process5deferIN5mesos8internal5slave5SlaveEiiSt12_PlaceholderILi1EES7_ILi2EEEENS1_9_DeferredIDTcl4bindadsrSt8functionIFvT0_T1_EEclcvSF__Efp1_fp2_EEEERKNS1_3PIDIT_EEMSJ_FvSC_SD_ET2_T3_EUliiE_E10_M_managerERSt9_Any_dataRKST_St18_Manager_operation
> [04:56:48]W: [Step 10/10] I1008 04:56:48.597909 25421 coordinator.cpp:238] Coordinator attempting to fill missing positions
> [04:56:48]W: [Step 10/10] I1008 04:56:48.598273 25423 replica.cpp:388] Replica received explicit promise request from __req_res__(4)@172.30.2.21:45167 for position 0 with proposal 2
> [04:56:48]W: [Step 10/10] @ 0x7f1b0d935b0b std::_Function_base::_Base_manager<>::_M_manager()
> [04:56:48]W: [Step 10/10] @ 0x7f1b0d8f2516 os::internal::configureSignal()
> [04:56:48]W: [Step 10/10] I1008 04:56:48.599318 25423 leveldb.cpp:341] Persisting action (8 bytes) to leveldb took 1.024957ms
> [04:56:48]W: [Step 10/10] I1008 04:56:48.599333 25423 replica.cpp:708] Persisted action NOP at position 0
> [04:56:48]W: [Step 10/10] I1008 04:56:48.599630 25428 replica.cpp:537] Replica received write request for position 0 from __req_res__(5)@172.30.2.21:45167
> [04:56:48]W: [Step 10/10] I1008 04:56:48.599660 25428 leveldb.cpp:436] Reading position from leveldb took 16893ns
> [04:56:48]W: [Step 10/10] @ 0x7f1b0d904022 mesos::internal::slave::Slave::initialize()
> [04:56:48]W: [Step 10/10] @ 0x7f1b0e0c6ed1 process::ProcessManager::resume()
> [04:56:48]W: [Step 10/10] @ 0x7f1b0e0c7187 _ZNSt6thread5_ImplISt12_Bind_simpleIFZN7process14ProcessManager12init_threadsEvEUt_vEEE6_M_runEv
> [04:56:48]W: [Step 10/10] @ 0x7f1b0c726a60 (unknown)
> [04:56:48]W: [Step 10/10] @ 0x7f1b0bf43182 start_thread
> [04:56:48]W: [Step 10/10] @ 0x7f1b0bc7047d (unknown)
> {code}
> This is seen specifically in Ubuntu 14.04
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)