You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@mesos.apache.org by "Chun-Hung Hsiao (JIRA)" <ji...@apache.org> on 2019/07/09 04:20:00 UTC
[jira] [Assigned] (MESOS-9260)
StorageLocalResourceProviderTest.ReconcileUnacknowledgedTerminalOperation
is flaky
[ https://issues.apache.org/jira/browse/MESOS-9260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chun-Hung Hsiao reassigned MESOS-9260:
--------------------------------------
Assignee: Chun-Hung Hsiao
> StorageLocalResourceProviderTest.ReconcileUnacknowledgedTerminalOperation is flaky
> ----------------------------------------------------------------------------------
>
> Key: MESOS-9260
> URL: https://issues.apache.org/jira/browse/MESOS-9260
> Project: Mesos
> Issue Type: Bug
> Reporter: Benno Evers
> Assignee: Chun-Hung Hsiao
> Priority: Major
> Labels: flaky-test
>
> Observed in an internal CI system:
> {noformat}
> ../../src/tests/storage_local_resource_provider_tests.cpp:4286
> Failed to wait 15secs for offers
> {noformat}
> Full log:
> {noformat}
> [ RUN ] StorageLocalResourceProviderTest.ReconcileUnacknowledgedTerminalOperation
> I0926 00:01:11.465590 11812 cluster.cpp:173] Creating default 'local' authorizer
> I0926 00:01:11.466861 1833 master.cpp:413] Master f0a77094-641d-45e5-a890-46f2c2d8202c (ip-172-16-10-227.ec2.internal) started on 172.16.10.227:43103
> I0926 00:01:11.466879 1833 master.cpp:416] Flags at startup: --acls="" --agent_ping_timeout="15secs" --agent_reregister_timeout="10mins" --allocation_interval="50ms" --allocator="hierarchical" --authenticate_agents="true" --authenticate_frameworks="true" --authenticate_http_frameworks="true" --authenticate_http_readonly="true" --authenticate_http_readwrite="true" --authentication_v0_timeout="15secs" --authenticators="crammd5" --authorizers="local" --credentials="/tmp/r7LBlp/credentials" --filter_gpu_resources="true" --framework_sorter="drf" --help="false" --hostname_lookup="true" --http_authenticators="basic" --http_framework_authenticators="basic" --initialize_driver_logging="true" --log_auto_initialize="true" --logbufsecs="0" --logging_level="INFO" --max_agent_ping_timeouts="5" --max_completed_frameworks="50" --max_completed_tasks_per_framework="1000" --max_unreachable_tasks_per_framework="1000" --memory_profiling="false" --min_allocatable_resources="cpus:0.01|mem:32" --port="5050" --quiet="false" --recovery_agent_removal_limit="100%" --registry="in_memory" --registry_fetch_timeout="1mins" --registry_gc_interval="15mins" --registry_max_agent_age="2weeks" --registry_max_agent_count="102400" --registry_store_timeout="100secs" --registry_strict="false" --require_agent_domain="false" --role_sorter="drf" --root_submissions="true" --version="false" --webui_dir="/usr/local/share/mesos/webui" --work_dir="/tmp/r7LBlp/master" --zk_session_timeout="10secs"
> I0926 00:01:11.467025 1833 master.cpp:465] Master only allowing authenticated frameworks to register
> I0926 00:01:11.467034 1833 master.cpp:471] Master only allowing authenticated agents to register
> I0926 00:01:11.467041 1833 master.cpp:477] Master only allowing authenticated HTTP frameworks to register
> I0926 00:01:11.467047 1833 credentials.hpp:37] Loading credentials for authentication from '/tmp/r7LBlp/credentials'
> I0926 00:01:11.467128 1833 master.cpp:521] Using default 'crammd5' authenticator
> I0926 00:01:11.467170 1833 http.cpp:1037] Creating default 'basic' HTTP authenticator for realm 'mesos-master-readonly'
> I0926 00:01:11.467207 1833 http.cpp:1037] Creating default 'basic' HTTP authenticator for realm 'mesos-master-readwrite'
> I0926 00:01:11.467229 1833 http.cpp:1037] Creating default 'basic' HTTP authenticator for realm 'mesos-master-scheduler'
> I0926 00:01:11.467248 1833 master.cpp:602] Authorization enabled
> I0926 00:01:11.467972 1833 hierarchical.cpp:182] Initialized hierarchical allocator process
> I0926 00:01:11.468001 1833 whitelist_watcher.cpp:77] No whitelist given
> I0926 00:01:11.468333 1833 master.cpp:2083] Elected as the leading master!
> I0926 00:01:11.468343 1833 master.cpp:1638] Recovering from registrar
> I0926 00:01:11.468374 1833 registrar.cpp:339] Recovering registrar
> I0926 00:01:11.468508 1833 registrar.cpp:383] Successfully fetched the registry (0B) in 0ns
> I0926 00:01:11.468541 1833 registrar.cpp:487] Applied 1 operations in 8882ns; attempting to update the registry
> I0926 00:01:11.468659 1833 registrar.cpp:544] Successfully updated the registry in 0ns
> I0926 00:01:11.468688 1833 registrar.cpp:416] Successfully recovered registrar
> I0926 00:01:11.468765 1833 master.cpp:1752] Recovered 0 agents from the registry (176B); allowing 10mins for agents to reregister
> I0926 00:01:11.468796 1833 hierarchical.cpp:220] Skipping recovery of hierarchical allocator: nothing to recover
> W0926 00:01:11.470585 11812 process.cpp:2810] Attempted to spawn already running process files@172.16.10.227:43103
> I0926 00:01:11.470960 11812 containerizer.cpp:305] Using isolation { cgroups/cpu, filesystem/posix, environment_secret, cgroups/mem, network/cni }
> I0926 00:01:11.472955 11812 linux_launcher.cpp:144] Using /sys/fs/cgroup/freezer as the freezer hierarchy for the Linux launcher
> I0926 00:01:11.473356 11812 provisioner.cpp:298] Using default backend 'overlay'
> I0926 00:01:11.478726 11812 cluster.cpp:485] Creating default 'local' authorizer
> I0926 00:01:11.479349 1840 slave.cpp:267] Mesos agent started on (771)@172.16.10.227:43103
> I0926 00:01:11.479365 1840 slave.cpp:268] Flags at startup: --acls="" --appc_simple_discovery_uri_prefix="http://" --appc_store_dir="/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_06lIVO/store/appc" --authenticate_http_executors="true" --authenticate_http_readonly="true" --authenticate_http_readwrite="true" --authenticatee="crammd5" --authentication_backoff_factor="1secs" --authentication_timeout_max="1mins" --authentication_timeout_min="5secs" --authorizer="local" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_destroy_timeout="1mins" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos_test_a71fca40-7489-4600-b9b0-b37c10625cff" --container_disk_watch_interval="15secs" --containerizers="mesos" --credential="/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_06lIVO/credential" --default_role="*" --disallow_sharing_agent_pid_namespace="false" --disk_profile_adaptor="org_apache_mesos_UriDiskProfileAdaptor" --disk_watch_interval="1mins" --docker="docker" --docker_kill_orphans="true" --docker_registry="https://registry-1.docker.io" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_06lIVO/store/docker" --docker_volume_checkpoint_dir="/var/run/mesos/isolators/docker/volume" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_reregistration_timeout="2secs" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_06lIVO/fetch" --fetcher_cache_size="2GB" --fetcher_stall_timeout="1mins" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --gc_non_executor_container_sandboxes="false" --help="false" --hostname_lookup="true" --http_command_executor="false" --http_credentials="/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_06lIVO/http_credentials" --http_heartbeat_interval="30secs" --initialize_driver_logging="true" --isolation="cgroups/cpu,cgroups/mem" --jwt_secret_key="/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_06lIVO/jwt_secret_key" --launcher="linux" --launcher_dir="/home/ubuntu/workspace/mesos/Mesos_CI-build/FLAG/SSL/label/mesos-ec2-ubuntu-16.04/mesos/build/src" --logbufsecs="0" --logging_level="INFO" --max_completed_executors_per_framework="150" --memory_profiling="false" --network_cni_metrics="true" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --port="5051" --qos_correction_interval_min="0ns" --quiet="false" --reconfiguration_policy="equal" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="10ms" --resource_provider_config_dir="/tmp/r7LBlp/resource_provider_configs" --resources="cpus:2;gpus:0;mem:1024;disk:1024;ports:[31000-32000]" --revocable_cpu_low_priority="true" --runtime_dir="/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_06lIVO" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_enable_support="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_YdUE86" --zk_session_timeout="10secs"
> I0926 00:01:11.479526 1840 credentials.hpp:86] Loading credential for authentication from '/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_06lIVO/credential'
> I0926 00:01:11.479574 1840 slave.cpp:300] Agent using credential for: test-principal
> I0926 00:01:11.479584 1840 credentials.hpp:37] Loading credentials for authentication from '/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_06lIVO/http_credentials'
> I0926 00:01:11.479640 1840 http.cpp:1037] Creating default 'basic' HTTP authenticator for realm 'mesos-agent-executor'
> I0926 00:01:11.479673 1840 http.cpp:1058] Creating default 'jwt' HTTP authenticator for realm 'mesos-agent-executor'
> I0926 00:01:11.479717 1840 http.cpp:1037] Creating default 'basic' HTTP authenticator for realm 'mesos-agent-readonly'
> I0926 00:01:11.479737 1840 http.cpp:1058] Creating default 'jwt' HTTP authenticator for realm 'mesos-agent-readonly'
> I0926 00:01:11.479763 1840 http.cpp:1037] Creating default 'basic' HTTP authenticator for realm 'mesos-agent-readwrite'
> I0926 00:01:11.479780 1840 http.cpp:1058] Creating default 'jwt' HTTP authenticator for realm 'mesos-agent-readwrite'
> I0926 00:01:11.479838 1840 disk_profile_adaptor.cpp:84] Creating disk profile adaptor module 'org_apache_mesos_UriDiskProfileAdaptor'
> I0926 00:01:11.480406 1840 slave.cpp:615] Agent resources: [{"name":"cpus","scalar":{"value":2.0},"type":"SCALAR"},{"name":"mem","scalar":{"value":1024.0},"type":"SCALAR"},{"name":"disk","scalar":{"value":1024.0},"type":"SCALAR"},{"name":"ports","ranges":{"range":[{"begin":31000,"end":32000}]},"type":"RANGES"}]
> I0926 00:01:11.480468 1840 slave.cpp:623] Agent attributes: [ ]
> I0926 00:01:11.480478 1840 slave.cpp:632] Agent hostname: ip-172-16-10-227.ec2.internal
> I0926 00:01:11.480512 1837 uri_disk_profile_adaptor.cpp:304] Updated disk profile mapping to 1 active profiles
> I0926 00:01:11.480752 1837 task_status_update_manager.cpp:181] Pausing sending task status updates
> I0926 00:01:11.480829 1837 state.cpp:66] Recovering state from '/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_YdUE86/meta'
> I0926 00:01:11.480952 1838 slave.cpp:6909] Finished recovering checkpointed state from '/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_YdUE86/meta', beginning agent recovery
> I0926 00:01:11.480989 1838 task_status_update_manager.cpp:207] Recovering task status update manager
> I0926 00:01:11.481066 1838 containerizer.cpp:727] Recovering Mesos containers
> I0926 00:01:11.481117 1838 linux_launcher.cpp:286] Recovering Linux launcher
> I0926 00:01:11.481254 1840 containerizer.cpp:1053] Recovering isolators
> I0926 00:01:11.481770 1837 containerizer.cpp:1092] Recovering provisioner
> I0926 00:01:11.481986 1835 provisioner.cpp:494] Provisioner recovery complete
> I0926 00:01:11.482172 1833 composing.cpp:339] Finished recovering all containerizers
> I0926 00:01:11.482262 1834 slave.cpp:7138] Recovering executors
> I0926 00:01:11.482285 1834 slave.cpp:7291] Finished recovery
> I0926 00:01:11.482527 1836 task_status_update_manager.cpp:181] Pausing sending task status updates
> I0926 00:01:11.482537 1837 slave.cpp:1254] New master detected at master@172.16.10.227:43103
> I0926 00:01:11.482563 1837 slave.cpp:1319] Detecting new master
> I0926 00:01:11.482641 1833 slave.cpp:1346] Authenticating with master master@172.16.10.227:43103
> I0926 00:01:11.482664 1833 slave.cpp:1355] Using default CRAM-MD5 authenticatee
> I0926 00:01:11.482738 1836 authenticatee.cpp:121] Creating new client SASL connection
> I0926 00:01:11.483110 1840 master.cpp:9653] Authenticating slave(771)@172.16.10.227:43103
> I0926 00:01:11.483165 1835 authenticator.cpp:414] Starting authentication session for crammd5-authenticatee(1491)@172.16.10.227:43103
> I0926 00:01:11.483230 1835 authenticator.cpp:98] Creating new server SASL connection
> I0926 00:01:11.483600 1835 authenticatee.cpp:213] Received SASL authentication mechanisms: CRAM-MD5
> I0926 00:01:11.483618 1835 authenticatee.cpp:239] Attempting to authenticate with mechanism 'CRAM-MD5'
> I0926 00:01:11.483647 1835 authenticator.cpp:204] Received SASL authentication start
> I0926 00:01:11.483681 1835 authenticator.cpp:326] Authentication requires more steps
> I0926 00:01:11.483709 1835 authenticatee.cpp:259] Received SASL authentication step
> I0926 00:01:11.483744 1835 authenticator.cpp:232] Received SASL authentication step
> I0926 00:01:11.483757 1835 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'ip-172-16-10-227.ec2.internal' server FQDN: 'ip-172-16-10-227.ec2.internal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false
> I0926 00:01:11.483767 1835 auxprop.cpp:181] Looking up auxiliary property '*userPassword'
> I0926 00:01:11.483778 1835 auxprop.cpp:181] Looking up auxiliary property '*cmusaslsecretCRAM-MD5'
> I0926 00:01:11.483786 1835 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'ip-172-16-10-227.ec2.internal' server FQDN: 'ip-172-16-10-227.ec2.internal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true
> I0926 00:01:11.483793 1835 auxprop.cpp:131] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true
> I0926 00:01:11.483798 1835 auxprop.cpp:131] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true
> I0926 00:01:11.483811 1835 authenticator.cpp:318] Authentication success
> I0926 00:01:11.483865 1840 authenticatee.cpp:299] Authentication success
> I0926 00:01:11.483928 1835 master.cpp:9685] Successfully authenticated principal 'test-principal' at slave(771)@172.16.10.227:43103
> I0926 00:01:11.483942 1839 authenticator.cpp:432] Authentication session cleanup for crammd5-authenticatee(1491)@172.16.10.227:43103
> I0926 00:01:11.484007 1833 slave.cpp:1446] Successfully authenticated with master master@172.16.10.227:43103
> I0926 00:01:11.484109 1833 slave.cpp:1877] Will retry registration in 14.703994ms if necessary
> I0926 00:01:11.484150 1835 master.cpp:6605] Received register agent message from slave(771)@172.16.10.227:43103 (ip-172-16-10-227.ec2.internal)
> I0926 00:01:11.484203 1835 master.cpp:3964] Authorizing agent providing resources 'cpus:2; mem:1024; disk:1024; ports:[31000-32000]' with principal 'test-principal'
> I0926 00:01:11.484340 1840 master.cpp:6672] Authorized registration of agent at slave(771)@172.16.10.227:43103 (ip-172-16-10-227.ec2.internal)
> I0926 00:01:11.484375 1840 master.cpp:6787] Registering agent at slave(771)@172.16.10.227:43103 (ip-172-16-10-227.ec2.internal) with id f0a77094-641d-45e5-a890-46f2c2d8202c-S0
> I0926 00:01:11.484493 1840 registrar.cpp:487] Applied 1 operations in 38466ns; attempting to update the registry
> I0926 00:01:11.484637 1835 registrar.cpp:544] Successfully updated the registry in 0ns
> I0926 00:01:11.484691 1834 master.cpp:6835] Admitted agent f0a77094-641d-45e5-a890-46f2c2d8202c-S0 at slave(771)@172.16.10.227:43103 (ip-172-16-10-227.ec2.internal)
> I0926 00:01:11.484822 1834 master.cpp:6880] Registered agent f0a77094-641d-45e5-a890-46f2c2d8202c-S0 at slave(771)@172.16.10.227:43103 (ip-172-16-10-227.ec2.internal) with cpus:2; mem:1024; disk:1024; ports:[31000-32000]
> I0926 00:01:11.484916 1834 hierarchical.cpp:601] Added agent f0a77094-641d-45e5-a890-46f2c2d8202c-S0 (ip-172-16-10-227.ec2.internal) with cpus:2; mem:1024; disk:1024; ports:[31000-32000] (allocated: {})
> I0926 00:01:11.484963 1834 hierarchical.cpp:1564] Performed allocation for 1 agents in 9501ns
> I0926 00:01:11.484994 1834 slave.cpp:1479] Registered with master master@172.16.10.227:43103; given agent ID f0a77094-641d-45e5-a890-46f2c2d8202c-S0
> I0926 00:01:11.485126 1834 slave.cpp:1499] Checkpointing SlaveInfo to '/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_YdUE86/meta/slaves/f0a77094-641d-45e5-a890-46f2c2d8202c-S0/slave.info'
> I0926 00:01:11.485360 1835 task_status_update_manager.cpp:188] Resuming sending task status updates
> I0926 00:01:11.485404 1834 slave.cpp:1548] Forwarding agent update {"operations":{},"resource_version_uuid":{"value":"TUGtB/13S1q/oBij0hB5rA=="},"slave_id":{"value":"f0a77094-641d-45e5-a890-46f2c2d8202c-S0"},"update_oversubscribed_resources":false}
> I0926 00:01:11.485775 1836 master.cpp:7939] Ignoring update on agent f0a77094-641d-45e5-a890-46f2c2d8202c-S0 at slave(771)@172.16.10.227:43103 (ip-172-16-10-227.ec2.internal) as it reports no changes
> I0926 00:01:11.487349 1836 process.cpp:3569] Handling HTTP event for process 'slave(771)' with path: '/slave(771)/api/v1'
> I0926 00:01:11.487823 1840 http.cpp:1177] HTTP POST for /slave(771)/api/v1 from 172.16.10.227:35910
> I0926 00:01:11.487898 1840 http.cpp:2074] Processing GET_CONTAINERS call
> I0926 00:01:11.490869 1836 container_daemon.cpp:121] Launching container 'org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE'
> I0926 00:01:11.491497 1833 process.cpp:3569] Handling HTTP event for process 'slave(771)' with path: '/slave(771)/api/v1'
> I0926 00:01:11.491904 1833 http.cpp:1177] HTTP POST for /slave(771)/api/v1 from 172.16.10.227:35912
> I0926 00:01:11.492103 1833 http.cpp:2440] Processing LAUNCH_CONTAINER call for container 'org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE'
> I0926 00:01:11.492333 1833 http.cpp:2556] Creating sandbox '/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_YdUE86/containers/org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE'
> I0926 00:01:11.493259 1836 containerizer.cpp:1280] Starting container org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
> I0926 00:01:11.493693 1836 containerizer.cpp:1446] Checkpointed ContainerConfig at '/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_06lIVO/containers/org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE/config'
> I0926 00:01:11.493710 1836 containerizer.cpp:3118] Transitioning the state of container org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE from PROVISIONING to PREPARING
> I0926 00:01:11.493806 1834 cgroups.cpp:429] Creating cgroup at '/sys/fs/cgroup/cpu,cpuacct/mesos_test_a71fca40-7489-4600-b9b0-b37c10625cff/org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE' for container org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
> I0926 00:01:11.494220 1834 cgroups.cpp:429] Creating cgroup at '/sys/fs/cgroup/memory/mesos_test_a71fca40-7489-4600-b9b0-b37c10625cff/org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE' for container org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
> I0926 00:01:11.494746 1834 memory.cpp:478] Started listening for OOM events for container org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
> I0926 00:01:11.494796 1834 memory.cpp:590] Started listening on 'low' memory pressure events for container org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
> I0926 00:01:11.494828 1834 memory.cpp:590] Started listening on 'medium' memory pressure events for container org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
> I0926 00:01:11.494858 1834 memory.cpp:590] Started listening on 'critical' memory pressure events for container org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
> I0926 00:01:11.495095 1836 memory.cpp:198] Updated 'memory.soft_limit_in_bytes' to 1GB for container org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
> I0926 00:01:11.495132 1834 cpu.cpp:92] Updated 'cpu.shares' to 102 (cpus 0.1) for container org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
> I0926 00:01:11.495165 1836 memory.cpp:227] Updated 'memory.limit_in_bytes' to 1GB for container org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
> I0926 00:01:11.495867 1834 containerizer.cpp:1939] Launching 'mesos-containerizer' with flags '--help="false" --launch_info="{"command":{"arguments":["/home/ubuntu/workspace/mesos/Mesos_CI-build/FLAG/SSL/label/mesos-ec2-ubuntu-16.04/mesos/build/src/test-csi-plugin","--available_capacity=4GB","--volumes=","--work_dir=/tmp/r7LBlp/test_csi_plugin"],"shell":false,"value":"/home/ubuntu/workspace/mesos/Mesos_CI-build/FLAG/SSL/label/mesos-ec2-ubuntu-16.04/mesos/build/src/test-csi-plugin"},"environment":{"variables":[{"name":"MESOS_SANDBOX","type":"VALUE","value":"/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_YdUE86/containers/org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE"},{"name":"CSI_ENDPOINT","type":"VALUE","value":"unix:///tmp/mesos-csi-wGzvgE/endpoint.sock"}]},"task_environment":{},"working_directory":"/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_YdUE86/containers/org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE"}" --pipe_read="75" --pipe_write="76" --runtime_directory="/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_06lIVO/containers/org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE" --unshare_namespace_mnt="false"'
> I0926 00:01:11.496028 1840 linux_launcher.cpp:492] Launching container org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE and cloning with namespaces
> I0926 00:01:11.521980 1834 containerizer.cpp:3118] Transitioning the state of container org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE from PREPARING to ISOLATING
> I0926 00:01:11.523646 1839 containerizer.cpp:3118] Transitioning the state of container org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE from ISOLATING to FETCHING
> I0926 00:01:11.523715 1839 fetcher.cpp:369] Starting to fetch URIs for container: org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE, directory: /tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_YdUE86/containers/org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
> I0926 00:01:11.523888 1839 containerizer.cpp:3118] Transitioning the state of container org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE from FETCHING to RUNNING
> I0926 00:01:11.525076 1835 container_daemon.cpp:140] Invoking post-start hook for container 'org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE'
> I0926 00:01:11.525574 1835 hierarchical.cpp:1564] Performed allocation for 1 agents in 18948ns
> I0926 00:01:11.576836 1836 hierarchical.cpp:1564] Performed allocation for 1 agents in 19631ns
> I0926 00:01:11.627815 1835 hierarchical.cpp:1564] Performed allocation for 1 agents in 21050ns
> I0926 00:01:11.639616 6260 test_csi_plugin.cpp:302] ProbeRequest '{}'
> I0926 00:01:11.640031 1839 container_daemon.cpp:171] Waiting for container 'org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE'
> I0926 00:01:11.640367 6260 test_csi_plugin.cpp:274] GetPluginInfoRequest '{}'
> I0926 00:01:11.640795 1836 process.cpp:3569] Handling HTTP event for process 'slave(771)' with path: '/slave(771)/api/v1'
> I0926 00:01:11.640939 1836 provider.cpp:2178] Node plugin loaded: {"name":"org.apache.mesos.csi.test","vendorVersion":"1.8.0"}
> I0926 00:01:11.641376 1834 http.cpp:1177] HTTP POST for /slave(771)/api/v1 from 172.16.10.227:35914
> I0926 00:01:11.641413 6261 test_csi_plugin.cpp:288] GetPluginCapabilitiesRequest '{}'
> I0926 00:01:11.641479 1834 http.cpp:2670] Processing WAIT_CONTAINER call for container 'org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE'
> I0926 00:01:11.641993 6261 test_csi_plugin.cpp:274] GetPluginInfoRequest '{}'
> I0926 00:01:11.642236 1839 provider.cpp:2219] Controller plugin loaded: {"name":"org.apache.mesos.csi.test","vendorVersion":"1.8.0"}
> I0926 00:01:11.642463 6261 test_csi_plugin.cpp:593] ControllerGetCapabilitiesRequest '{}'
> I0926 00:01:11.642946 6261 test_csi_plugin.cpp:870] NodeGetCapabilitiesRequest '{}'
> I0926 00:01:11.643378 6261 test_csi_plugin.cpp:857] NodeGetIdRequest '{}'
> I0926 00:01:11.643676 1839 provider.cpp:661] Finished recovery for resource provider with type 'org.apache.mesos.rp.local.storage' and name 'test
> I0926 00:01:11.643731 1834 status_update_manager_process.hpp:379] Pausing operation status update manager
> I0926 00:01:11.643767 1839 http_connection.hpp:227] New endpoint detected at http://172.16.10.227:43103/slave(771)/api/v1/resource_provider
> I0926 00:01:11.644228 1838 http_connection.hpp:283] Connected with the remote endpoint at http://172.16.10.227:43103/slave(771)/api/v1/resource_provider
> I0926 00:01:11.644340 1839 provider.cpp:524] Connected to resource provider manager
> I0926 00:01:11.644498 1835 http_connection.hpp:131] Sending 1 call to http://172.16.10.227:43103/slave(771)/api/v1/resource_provider
> I0926 00:01:11.644778 1837 process.cpp:3569] Handling HTTP event for process 'slave(771)' with path: '/slave(771)/api/v1/resource_provider'
> I0926 00:01:11.645036 1840 http.cpp:1177] HTTP POST for /slave(771)/api/v1/resource_provider from 172.16.10.227:35918
> I0926 00:01:11.645167 1840 manager.cpp:736] Subscribing resource provider {"default_reservations":[{"role":"storage","type":"DYNAMIC"}],"name":"test","storage":{"plugin":{"containers":[{"command":{"arguments":["/home/ubuntu/workspace/mesos/Mesos_CI-build/FLAG/SSL/label/mesos-ec2-ubuntu-16.04/mesos/build/src/test-csi-plugin","--available_capacity=4GB","--volumes=","--work_dir=/tmp/r7LBlp/test_csi_plugin"],"shell":false,"value":"/home/ubuntu/workspace/mesos/Mesos_CI-build/FLAG/SSL/label/mesos-ec2-ubuntu-16.04/mesos/build/src/test-csi-plugin"},"resources":[{"name":"cpus","scalar":{"value":0.1},"type":"SCALAR"},{"name":"mem","scalar":{"value":1024.0},"type":"SCALAR"}],"services":["CONTROLLER_SERVICE","NODE_SERVICE"]}],"name":"test_csi_plugin","type":"org.apache.mesos.csi.test"}},"type":"org.apache.mesos.rp.local.storage"}
> I0926 00:01:11.647924 1840 slave.cpp:7643] Handling resource provider message 'SUBSCRIBE: {"default_reservations":[{"role":"storage","type":"DYNAMIC"}],"id":{"value":"520ef77a-5d14-4c9f-8077-0a6c4ae0b0b2"},"name":"test","storage":{"plugin":{"containers":[{"command":{"arguments":["/home/ubuntu/workspace/mesos/Mesos_CI-build/FLAG/SSL/label/mesos-ec2-ubuntu-16.04/mesos/build/src/test-csi-plugin","--available_capacity=4GB","--volumes=","--work_dir=/tmp/r7LBlp/test_csi_plugin"],"shell":false,"value":"/home/ubuntu/workspace/mesos/Mesos_CI-build/FLAG/SSL/label/mesos-ec2-ubuntu-16.04/mesos/build/src/test-csi-plugin"},"resources":[{"name":"cpus","scalar":{"value":0.1},"type":"SCALAR"},{"name":"mem","scalar":{"value":1024.0},"type":"SCALAR"}],"services":["CONTROLLER_SERVICE","NODE_SERVICE"]}],"name":"test_csi_plugin","type":"org.apache.mesos.csi.test"}},"type":"org.apache.mesos.rp.local.storage"}'
> I0926 00:01:11.648219 1837 provider.cpp:546] Received SUBSCRIBED event
> I0926 00:01:11.648238 1837 provider.cpp:1496] Subscribed with ID 520ef77a-5d14-4c9f-8077-0a6c4ae0b0b2
> I0926 00:01:11.648468 1836 status_update_manager_process.hpp:314] Recovering operation status update manager
> I0926 00:01:11.648780 6261 test_csi_plugin.cpp:531] ListVolumesRequest '{}'
> I0926 00:01:11.649070 1839 provider.cpp:3519] Sending UPDATE_STATE call with resources '{}' and 0 operations to agent f0a77094-641d-45e5-a890-46f2c2d8202c-S0
> I0926 00:01:11.649124 1839 provider.cpp:1110] Resource provider 520ef77a-5d14-4c9f-8077-0a6c4ae0b0b2 is in READY state
> I0926 00:01:11.649148 1834 http_connection.hpp:131] Sending 3 call to http://172.16.10.227:43103/slave(771)/api/v1/resource_provider
> I0926 00:01:11.649207 1839 status_update_manager_process.hpp:385] Resuming operation status update manager
> I0926 00:01:11.649250 1835 provider.cpp:1422] Updating profiles { test } for resource provider 520ef77a-5d14-4c9f-8077-0a6c4ae0b0b2
> I0926 00:01:11.649550 1837 process.cpp:3569] Handling HTTP event for process 'slave(771)' with path: '/slave(771)/api/v1/resource_provider'
> I0926 00:01:11.649752 1837 http.cpp:1177] HTTP POST for /slave(771)/api/v1/resource_provider from 172.16.10.227:35916
> I0926 00:01:11.649860 1837 manager.cpp:917] Received UPDATE_STATE call with resources '[]' and 0 operations from resource provider 520ef77a-5d14-4c9f-8077-0a6c4ae0b0b2
> I0926 00:01:11.649793 6261 test_csi_plugin.cpp:559] GetCapacityRequest '{"volumeCapabilities":[{"mount":{},"accessMode":{"mode":"SINGLE_NODE_WRITER"}}]}'
> I0926 00:01:11.649943 1837 slave.cpp:7643] Handling resource provider message 'UPDATE_STATE: 520ef77a-5d14-4c9f-8077-0a6c4ae0b0b2 {}'
> I0926 00:01:11.649966 1837 slave.cpp:7763] Forwarding new total resources cpus:2; mem:1024; disk:1024; ports:[31000-32000]
> I0926 00:01:11.650372 1837 provider.cpp:1297] Removing '{}' and adding 'disk(reservations: [(DYNAMIC,storage)])[RAW(,test)]:4096' to the total resources
> I0926 00:01:11.650480 1833 hierarchical.cpp:740] Grew agent f0a77094-641d-45e5-a890-46f2c2d8202c-S0 by {} (total), { } (used)
> I0926 00:01:11.650538 1833 hierarchical.cpp:697] Agent f0a77094-641d-45e5-a890-46f2c2d8202c-S0 (ip-172-16-10-227.ec2.internal) updated with total resources cpus:2; mem:1024; disk:1024; ports:[31000-32000]
> I0926 00:01:11.650691 1837 provider.cpp:3519] Sending UPDATE_STATE call with resources 'disk(reservations: [(DYNAMIC,storage)])[RAW(,test)]:4096' and 0 operations to agent f0a77094-641d-45e5-a890-46f2c2d8202c-S0
> I0926 00:01:11.650811 1833 http_connection.hpp:131] Sending 3 call to http://172.16.10.227:43103/slave(771)/api/v1/resource_provider
> I0926 00:01:11.651093 1840 process.cpp:3569] Handling HTTP event for process 'slave(771)' with path: '/slave(771)/api/v1/resource_provider'
> I0926 00:01:11.651756 11812 scheduler.cpp:189] Version: 1.8.0
> I0926 00:01:11.652252 1838 scheduler.cpp:355] Using default 'basic' HTTP authenticatee
> I0926 00:01:11.652364 1836 scheduler.cpp:538] New master detected at master@172.16.10.227:43103
> I0926 00:01:11.652377 1836 scheduler.cpp:547] Waiting for 0ns before initiating a re-(connection) attempt with the master
> I0926 00:01:11.652827 1835 scheduler.cpp:429] Connected with the master at http://172.16.10.227:43103/master/api/v1/scheduler
> I0926 00:01:11.653015 1838 scheduler.cpp:248] Sending SUBSCRIBE call to http://172.16.10.227:43103/master/api/v1/scheduler
> I0926 00:01:11.653323 1833 process.cpp:3569] Handling HTTP event for process 'master' with path: '/master/api/v1/scheduler'
> I0926 00:01:11.653650 1840 http.cpp:1177] HTTP POST for /master/api/v1/scheduler from 172.16.10.227:35922
> I0926 00:01:11.653709 1840 master.cpp:2502] Received subscription request for HTTP framework 'default'
> I0926 00:01:11.653733 1840 master.cpp:2155] Authorizing framework principal 'test-principal' to receive offers for roles '{ storage }'
> I0926 00:01:11.653841 1834 master.cpp:2637] Subscribing framework 'default' with checkpointing disabled and capabilities [ MULTI_ROLE, RESERVATION_REFINEMENT ]
> I0926 00:01:11.654402 1834 master.cpp:9883] Adding framework f0a77094-641d-45e5-a890-46f2c2d8202c-0000 (default) with roles { } suppressed
> I0926 00:01:11.654527 1837 hierarchical.cpp:306] Added framework f0a77094-641d-45e5-a890-46f2c2d8202c-0000
> I0926 00:01:11.654933 1836 scheduler.cpp:845] Enqueuing event SUBSCRIBED received from http://172.16.10.227:43103/master/api/v1/scheduler
> I0926 00:01:11.654948 1837 hierarchical.cpp:1564] Performed allocation for 1 agents in 140494ns
> I0926 00:01:11.655035 1835 master.cpp:9468] Sending offers [ f0a77094-641d-45e5-a890-46f2c2d8202c-O0 ] to framework f0a77094-641d-45e5-a890-46f2c2d8202c-0000 (default)
> I0926 00:01:11.689805 1839 http.cpp:1177] HTTP POST for /slave(771)/api/v1/resource_provider from 172.16.10.227:35916
> I0926 00:01:13.959466 1834 manager.cpp:917] Received UPDATE_STATE call with resources '[{"disk":{"source":{"profile":"test","type":"RAW"}},"name":"disk","provider_id":{"value":"520ef77a-5d14-4c9f-8077-0a6c4ae0b0b2"},"reservations":[{"role":"storage","type":"DYNAMIC"}],"scalar":{"value":4096.0},"type":"SCALAR"}]' and 0 operations from resource provider 520ef77a-5d14-4c9f-8077-0a6c4ae0b0b2
> I0926 00:01:13.960250 1839 slave.cpp:7643] Handling resource provider message 'UPDATE_STATE: 520ef77a-5d14-4c9f-8077-0a6c4ae0b0b2 disk(reservations: [(DYNAMIC,storage)])[RAW(,test)]:4096'
> I0926 00:01:13.960312 1839 slave.cpp:7763] Forwarding new total resources cpus:2; mem:1024; disk:1024; ports:[31000-32000]; disk(reservations: [(DYNAMIC,storage)])[RAW(,test)]:4096
> I0926 00:01:13.960785 1838 hierarchical.cpp:697] Agent f0a77094-641d-45e5-a890-46f2c2d8202c-S0 (ip-172-16-10-227.ec2.internal) updated with total resources cpus:2; mem:1024; disk:1024; ports:[31000-32000]; disk(reservations: [(DYNAMIC,storage)])[RAW(,test)]:4096
> I0926 00:01:13.960876 1838 hierarchical.cpp:1564] Performed allocation for 1 agents in 40437ns
> I0926 00:01:13.965555 1836 scheduler.cpp:845] Enqueuing event HEARTBEAT received from http://172.16.10.227:43103/master/api/v1/scheduler
> I0926 00:01:13.966929 1836 scheduler.cpp:845] Enqueuing event OFFERS received from http://172.16.10.227:43103/master/api/v1/scheduler
> I0926 00:01:13.967149 1838 scheduler.cpp:248] Sending DECLINE call to http://172.16.10.227:43103/master/api/v1/scheduler
> I0926 00:01:13.967648 1835 process.cpp:3569] Handling HTTP event for process 'master' with path: '/master/api/v1/scheduler'
> I0926 00:01:13.967850 1837 hierarchical.cpp:1564] Performed allocation for 1 agents in 33598ns
> I0926 00:01:13.968111 1833 http.cpp:1177] HTTP POST for /master/api/v1/scheduler from 172.16.10.227:35920
> I0926 00:01:13.968158 1833 master.cpp:5850] Processing DECLINE call for offers: [ f0a77094-641d-45e5-a890-46f2c2d8202c-O0 ] for framework f0a77094-641d-45e5-a890-46f2c2d8202c-0000 (default)
> I0926 00:01:13.968245 1833 master.cpp:11462] Removing offer f0a77094-641d-45e5-a890-46f2c2d8202c-O0
> I0926 00:01:13.968375 1833 hierarchical.cpp:1236] Recovered cpus(allocated: storage):2; mem(allocated: storage):1024; disk(allocated: storage):1024; ports(allocated: storage):[31000-32000] (total: cpus:2; mem:1024; disk:1024; ports:[31000-32000]; disk(reservations: [(DYNAMIC,storage)])[RAW(,test)]:4096, allocated: {}) on agent f0a77094-641d-45e5-a890-46f2c2d8202c-S0 from framework f0a77094-641d-45e5-a890-46f2c2d8202c-0000
> I0926 00:01:13.968408 1833 hierarchical.cpp:1282] Framework f0a77094-641d-45e5-a890-46f2c2d8202c-0000 filtered agent f0a77094-641d-45e5-a890-46f2c2d8202c-S0 for 5secs
> ../../src/tests/storage_local_resource_provider_tests.cpp:4286: Failure
> Failed to wait 15secs for offers
> I0926 00:01:28.977944 1833 master.cpp:1366] Framework f0a77094-641d-45e5-a890-46f2c2d8202c-0000 (default) disconnected
> I0926 00:01:28.977985 1833 master.cpp:3230] Deactivating framework f0a77094-641d-45e5-a890-46f2c2d8202c-0000 (default)
> I0926 00:01:28.978009 1833 master.cpp:3207] Disconnecting framework f0a77094-641d-45e5-a890-46f2c2d8202c-0000 (default)
> I0926 00:01:28.978024 1833 master.cpp:1381] Giving framework f0a77094-641d-45e5-a890-46f2c2d8202c-0000 (default) 0ns to failover
> I0926 00:01:28.978085 1833 hierarchical.cpp:420] Deactivated framework f0a77094-641d-45e5-a890-46f2c2d8202c-0000
> I0926 00:01:28.978204 1837 master.cpp:9261] Framework failover timeout, removing framework f0a77094-641d-45e5-a890-46f2c2d8202c-0000 (default)
> I0926 00:01:28.978226 1837 master.cpp:10197] Removing framework f0a77094-641d-45e5-a890-46f2c2d8202c-0000 (default)
> I0926 00:01:28.978302 1840 slave.cpp:3896] Asked to shut down framework f0a77094-641d-45e5-a890-46f2c2d8202c-0000 by master@172.16.10.227:43103
> I0926 00:01:28.978327 1840 slave.cpp:3911] Cannot shut down unknown framework f0a77094-641d-45e5-a890-46f2c2d8202c-0000
> I0926 00:01:28.978341 1837 hierarchical.cpp:359] Removed framework f0a77094-641d-45e5-a890-46f2c2d8202c-0000
> ../../src/tests/storage_local_resource_provider_tests.cpp:4266: Failure
> Actual function call count doesn't match EXPECT_CALL(*scheduler, offers(_, v1::scheduler::OffersHaveAnyResource(isRaw)))...
> Expected: to be called once
> Actual: never called - unsatisfied and active
> I0926 00:01:28.978569 11812 slave.cpp:909] Agent terminating
> I0926 00:01:28.978735 11812 manager.cpp:161] Terminating resource provider 520ef77a-5d14-4c9f-8077-0a6c4ae0b0b2
> E0926 00:01:28.979059 1836 http_connection.hpp:452] End-Of-File received
> I0926 00:01:28.979130 1836 http_connection.hpp:217] Re-detecting endpoint
> I0926 00:01:28.979238 1836 http_connection.hpp:227] New endpoint detected at http://172.16.10.227:43103/slave(771)/api/v1/resource_provider
> I0926 00:01:28.979274 1839 provider.cpp:536] Disconnected from resource provider manager
> I0926 00:01:28.979276 1836 http_connection.hpp:338] Ignoring disconnection attempt from stale connection
> I0926 00:01:28.979300 1836 http_connection.hpp:338] Ignoring disconnection attempt from stale connection
> I0926 00:01:28.979308 1839 status_update_manager_process.hpp:379] Pausing operation status update manager
> I0926 00:01:28.979518 1834 master.cpp:1251] Agent f0a77094-641d-45e5-a890-46f2c2d8202c-S0 at slave(771)@172.16.10.227:43103 (ip-172-16-10-227.ec2.internal) disconnected
> I0926 00:01:28.979540 1834 master.cpp:3267] Disconnecting agent f0a77094-641d-45e5-a890-46f2c2d8202c-S0 at slave(771)@172.16.10.227:43103 (ip-172-16-10-227.ec2.internal)
> I0926 00:01:28.979560 1834 master.cpp:3286] Deactivating agent f0a77094-641d-45e5-a890-46f2c2d8202c-S0 at slave(771)@172.16.10.227:43103 (ip-172-16-10-227.ec2.internal)
> I0926 00:01:28.979599 1834 hierarchical.cpp:795] Agent f0a77094-641d-45e5-a890-46f2c2d8202c-S0 deactivated
> I0926 00:01:28.979914 1839 containerizer.cpp:2455] Destroying container org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE in RUNNING state
> I0926 00:01:28.979934 1839 containerizer.cpp:3118] Transitioning the state of container org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE from RUNNING to DESTROYING
> I0926 00:01:28.980154 1839 linux_launcher.cpp:580] Asked to destroy container org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
> I0926 00:01:28.980207 1839 linux_launcher.cpp:622] Destroying cgroup '/sys/fs/cgroup/freezer/mesos_test_a71fca40-7489-4600-b9b0-b37c10625cff/org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE'
> I0926 00:01:28.980270 1840 http_connection.hpp:283] Connected with the remote endpoint at http://172.16.10.227:43103/slave(771)/api/v1/resource_provider
> I0926 00:01:28.980406 1840 hierarchical.cpp:1564] Performed allocation for 1 agents in 26143ns
> I0926 00:01:28.980520 1836 provider.cpp:524] Connected to resource provider manager
> I0926 00:01:28.980541 1838 cgroups.cpp:2838] Freezing cgroup /sys/fs/cgroup/freezer/mesos_test_a71fca40-7489-4600-b9b0-b37c10625cff/org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
> I0926 00:01:28.980742 1835 http_connection.hpp:131] Sending 1 call to http://172.16.10.227:43103/slave(771)/api/v1/resource_provider
> I0926 00:01:28.980916 1838 cgroups.cpp:1229] Successfully froze cgroup /sys/fs/cgroup/freezer/mesos_test_a71fca40-7489-4600-b9b0-b37c10625cff/org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE after 349184ns
> I0926 00:01:28.981250 1837 cgroups.cpp:2856] Thawing cgroup /sys/fs/cgroup/freezer/mesos_test_a71fca40-7489-4600-b9b0-b37c10625cff/org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
> I0926 00:01:28.981411 1837 cgroups.cpp:1258] Successfully thawed cgroup /sys/fs/cgroup/freezer/mesos_test_a71fca40-7489-4600-b9b0-b37c10625cff/org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE after 124928ns
> I0926 00:01:28.981528 1841 process.cpp:2735] Returning '404 Not Found' for '/slave(771)/api/v1/resource_provider'
> E0926 00:01:28.981875 1840 provider.cpp:1067] Failed to subscribe resource provider with type 'org.apache.mesos.rp.local.storage' and name 'test': Received '404 Not Found' ()
> I0926 00:01:29.031200 1835 hierarchical.cpp:1564] Performed allocation for 1 agents in 26888ns
> I0926 00:01:29.081390 1836 hierarchical.cpp:1564] Performed allocation for 1 agents in 21841ns
> I0926 00:01:29.081869 1836 containerizer.cpp:2957] Container org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE has exited
> I0926 00:01:29.083161 1839 provisioner.cpp:597] Ignoring destroy request for unknown container org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
> I0926 00:01:29.083875 1834 container_daemon.cpp:189] Invoking post-stop hook for container 'org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE'
> I0926 00:01:29.084064 1834 container_daemon.cpp:121] Launching container 'org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE'
> I0926 00:01:29.086195 1841 process.cpp:2735] Returning '404 Not Found' for '/slave(771)/api/v1'
> E0926 00:01:29.086671 1833 container_daemon.cpp:150] Failed to launch container 'org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE': Failed to launch container 'org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE': Unexpected response '404 Not Found' ()
> I0926 00:01:29.099272 11812 master.cpp:1093] Master terminating
> I0926 00:01:29.099445 1836 hierarchical.cpp:637] Removed agent f0a77094-641d-45e5-a890-46f2c2d8202c-S0
> I0926 00:01:29.113350 1837 cgroups.cpp:2838] Freezing cgroup /sys/fs/cgroup/freezer/mesos_test_a71fca40-7489-4600-b9b0-b37c10625cff
> I0926 00:01:29.113492 1837 cgroups.cpp:1229] Successfully froze cgroup /sys/fs/cgroup/freezer/mesos_test_a71fca40-7489-4600-b9b0-b37c10625cff after 99072ns
> I0926 00:01:29.113646 1837 cgroups.cpp:2856] Thawing cgroup /sys/fs/cgroup/freezer/mesos_test_a71fca40-7489-4600-b9b0-b37c10625cff
> I0926 00:01:29.113750 1837 cgroups.cpp:1258] Successfully thawed cgroup /sys/fs/cgroup/freezer/mesos_test_a71fca40-7489-4600-b9b0-b37c10625cff after 72960ns
> [ FAILED ] StorageLocalResourceProviderTest.ReconcileUnacknowledgedTerminalOperation (17703 ms)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)