You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@mesos.apache.org by "Chun-Hung Hsiao (JIRA)" <ji...@apache.org> on 2018/11/20 05:42:01 UTC
[jira] [Assigned] (MESOS-9395) Check failure on
`StorageLocalResourceProviderProcess::applyCreateDisk`.
[ https://issues.apache.org/jira/browse/MESOS-9395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chun-Hung Hsiao reassigned MESOS-9395:
--------------------------------------
Assignee: Chun-Hung Hsiao
> Check failure on `StorageLocalResourceProviderProcess::applyCreateDisk`.
> ------------------------------------------------------------------------
>
> Key: MESOS-9395
> URL: https://issues.apache.org/jira/browse/MESOS-9395
> Project: Mesos
> Issue Type: Bug
> Components: resource provider
> Affects Versions: 1.7.0
> Reporter: Alexander Rukletsov
> Assignee: Chun-Hung Hsiao
> Priority: Critical
> Labels: mesosphere, storage
>
> Observed the following agent failure on one of our staging clusters:
> {noformat}
> Nov 16 11:57:24 int-mountvolumeagent2-soak112s.testing.mesosphe.re mesos-agent[26663]: I1116 11:57:24.641331 26684 http.cpp:1799] Processing GET_AGENT call
> Nov 16 11:57:24 int-mountvolumeagent2-soak112s.testing.mesosphe.re mesos-agent[26663]: I1116 11:57:24.650429 26679 http.cpp:1117] HTTP POST for /slave(1)/api/v1/resource_provider from 172.31.8.65:57790
> Nov 16 11:57:24 int-mountvolumeagent2-soak112s.testing.mesosphe.re mesos-agent[26663]: I1116 11:57:24.650629 26679 manager.cpp:672] Subscribing resource provider {"attributes":[{"name":"lvm-vg-name","text":{"value":"lvm-double-1540383639"},"type":"SCALAR"},{"name":"dss-asset-id","text":{"value":"6AbZV6W2DrK4YgcIR3ICVo"},"type":"SCALAR"}],"default_reservations":[{"principal":"storage-principal","role":"dcos-storage","type":"DYNAMIC"}],"id":{"value":"8326e931-41f2-4f45-9174-13fe35c19300"},"name":"rp_6AbZV6W2DrK4YgcIR3ICVo","storage":{"plugin":{"containers":[{"command":{"environment":{"variables":[{"name":"PATH","type":"VALUE","value":"/opt/mesosphere/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"},{"name":"LD_LIBRARY_PATH","type":"VALUE","value":"/opt/mesosphere/lib"},{"name":"CONTAINER_LOGGER_DESTINATION_TYPE","type":"VALUE","value":"journald+logrotate"},{"name":"CONTAINER_LOGGER_EXTRA_LABELS","type":"VALUE","value":"{\"CSI_PLUGIN\":\"csilvm\"}"}]},"shell":true,"uris":[{"executable":true,"extract":false,"value":"<possibly-sensitive>"}],"value":"echo \"a *:* rwm\" > /sys/fs/cgroup/devices`cat /proc/self/cgroup | grep devices | cut -d : -f 3`/devices.allow; exec ./csilvm -devices=/dev/xvdk,/dev/xvdj -volume-group=lvm-double-1540383639 -unix-addr-env=CSI_ENDPOINT -tag=6AbZV6W2DrK4YgcIR3ICVo"},"resources":[{"name":"cpus","scalar":{"value":0.1},"type":"SCALAR"},{"name":"mem","scalar":{"value":128.0},"type":"SCALAR"},{"name":"disk","scalar":{"value":10.0},"type":"SCALAR"}],"services":["CONTROLLER_SERVICE","NODE_SERVICE"]}],"name":"plugin_6AbZV6W2DrK4YgcIR3ICVo","type":"io.mesosphere.dcos.storage.csilvm"}},"type":"org.apache.mesos.rp.local.storage"}
> Nov 16 11:57:24 int-mountvolumeagent2-soak112s.testing.mesosphe.re mesos-agent[26663]: I1116 11:57:24.690474 26685 provider.cpp:546] Received SUBSCRIBED event
> Nov 16 11:57:24 int-mountvolumeagent2-soak112s.testing.mesosphe.re mesos-agent[26663]: I1116 11:57:24.690521 26685 provider.cpp:1492] Subscribed with ID 8326e931-41f2-4f45-9174-13fe35c19300
> Nov 16 11:57:24 int-mountvolumeagent2-soak112s.testing.mesosphe.re mesos-agent[26663]: I1116 11:57:24.690657 26681 status_update_manager_process.hpp:314] Recovering operation status update manager
> Nov 16 11:57:24 int-mountvolumeagent2-soak112s.testing.mesosphe.re mesos-agent[26663]: F1116 11:57:24.691496 26682 provider.cpp:3121] Check failed: resource.disk().source().has_profile() != resource.disk().source().has_id() (1 vs. 1)
> Nov 16 11:57:24 int-mountvolumeagent2-soak112s.testing.mesosphe.re mesos-agent[26663]: *** Check failure stack trace: ***
> Nov 16 11:57:24 int-mountvolumeagent2-soak112s.testing.mesosphe.re mesos-agent[26663]: @ 0x7fecb099e9fd google::LogMessage::Fail()
> Nov 16 11:57:24 int-mountvolumeagent2-soak112s.testing.mesosphe.re mesos-agent[26663]: @ 0x7fecb09a082d google::LogMessage::SendToLog()
> Nov 16 11:57:24 int-mountvolumeagent2-soak112s.testing.mesosphe.re mesos-agent[26663]: @ 0x7fecb099e5ec google::LogMessage::Flush()
> Nov 16 11:57:24 int-mountvolumeagent2-soak112s.testing.mesosphe.re mesos-agent[26663]: @ 0x7fecb09a1129 google::LogMessageFatal::~LogMessageFatal()
> Nov 16 11:57:24 int-mountvolumeagent2-soak112s.testing.mesosphe.re mesos-agent[26663]: @ 0x7fecb01654ca mesos::internal::StorageLocalResourceProviderProcess::applyCreateDisk()
> Nov 16 11:57:24 int-mountvolumeagent2-soak112s.testing.mesosphe.re mesos-agent[26663]: @ 0x7fecb017c683 mesos::internal::StorageLocalResourceProviderProcess::_applyOperation()
> Nov 16 11:57:24 int-mountvolumeagent2-soak112s.testing.mesosphe.re mesos-agent[26663]: @ 0x7fecb017d64a _ZZN5mesos8internal35StorageLocalResourceProviderProcess26reconcileOperationStatusesEvENKUlRKNS0_26StatusUpdateManagerProcessIN2id4UUIDENS0_27UpdateOperationStatusRecordENS0_28UpdateOperationStatusMessageEE5StateEE_clESA_
> Nov 16 11:57:24 int-mountvolumeagent2-soak112s.testing.mesosphe.re mesos-agent[26663]: @ 0x7fecb017dd21 _ZNO6lambda12CallableOnceIFN7process6FutureI7NothingEEvEE10CallableFnINS_8internal7PartialIZN5mesos8internal35StorageLocalResourceProviderProcess26reconcileOperationStatusesEvEUlRKNSB_26StatusUpdateManagerProcessIN2id4UUIDENSB_27UpdateOperationStatusRecordENSB_28UpdateOperationStatusMessageEE5StateEE_ISJ_EEEEclEv
> Nov 16 11:57:24 int-mountvolumeagent2-soak112s.testing.mesosphe.re mesos-agent[26663]: @ 0x7fecafa0ce97 _ZNO6lambda12CallableOnceIFvPN7process11ProcessBaseEEE10CallableFnINS_8internal7PartialIZNS1_8internal8DispatchINS1_6FutureI7NothingEEEclINS0_IFSD_vEEEEESD_RKNS1_4UPIDEOT_EUlSt10unique_ptrINS1_7PromiseISC_EESt14default_deleteISP_EEOSH_S3_E_JSS_SH_St12_PlaceholderILi1EEEEEEclEOS3_
> Nov 16 11:57:24 int-mountvolumeagent2-soak112s.testing.mesosphe.re mesos-agent[26663]: @ 0x7fecb08eec51 process::ProcessBase::consume()
> Nov 16 11:57:24 int-mountvolumeagent2-soak112s.testing.mesosphe.re mesos-agent[26663]: @ 0x7fecb09056cc process::ProcessManager::resume()
> Nov 16 11:57:24 int-mountvolumeagent2-soak112s.testing.mesosphe.re mesos-agent[26663]: @ 0x7fecb090b186 _ZNSt6thread5_ImplISt12_Bind_simpleIFZN7process14ProcessManager12init_threadsEvEUlvE_vEEE6_M_runEv
> Nov 16 11:57:24 int-mountvolumeagent2-soak112s.testing.mesosphe.re mesos-agent[26663]: @ 0x7fecad5d8070 (unknown)
> Nov 16 11:57:24 int-mountvolumeagent2-soak112s.testing.mesosphe.re mesos-agent[26663]: @ 0x7fecacdf6e25 start_thread
> Nov 16 11:57:24 int-mountvolumeagent2-soak112s.testing.mesosphe.re mesos-agent[26663]: @ 0x7fecacb20bad __clone
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)