You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Boris Stoyanov <bo...@shapeblue.com> on 2016/12/23 12:53:29 UTC
[DISCUSS][BUG] iscsi being attached but it's not writable
Hi all,
I’m moving this conversation here, to be picked up by a separate PR since we’ve merged the original 1854.
Pasting original content from github:
@mike-tutkowski<https://github.com/mike-tutkowski> @syed<https://github.com/syed>,
I've been investigating the test failure from test_primary_storage.py: test_01_primary_storage_iscsi, and we came to a conclusion that iscsi attaches but it's not writeable.
By any chance do you recall any change that might cause this?
Here's the management log failing to create a hearbeat volume:
2016-12-23 11:30:58,826 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-404:ctx-c95759e3) (logid:f96038aa) Setting up the heartbeat sr for host 10.2.2.33 and sr ca748d82-3668-9559-07ba-19ff94844ed0
2016-12-23 11:30:59,104 WARN [c.c.h.x.r.w.x.CitrixModifyStoragePoolCommandWrapper] (DirectAgent-404:ctx-c95759e3) (logid:f96038aa) ModifyStoragePoolCommand add XenAPIException:Unable to setup heartbeat
sr on SR ca748d82-3668-9559-07ba-19ff94844ed0 due to #6# Unable to create heartbeat volume hb-db2a4468-ef57-4936-9ccd-7b8f4c06d822 host:db2a4468-ef57-4936-9ccd-7b8f4c06d822 pool: 10.5.0.4/iqn.2016-10.l
ocal.marvin:tgt1/0
com.cloud.utils.exception.CloudRuntimeException: Unable to setup heartbeat sr on SR ca748d82-3668-9559-07ba-19ff94844ed0 due to #6# Unable to create heartbeat volume hb-db2a4468-ef57-4936-9ccd-7b8f4c06d
822
at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.setupHeartbeatSr(CitrixResourceBase.java:4596)
at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixModifyStoragePoolCommandWrapper.execute(CitrixModifyStoragePoolCommandWrapper.java:54)
at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixModifyStoragePoolCommandWrapper.execute(CitrixModifyStoragePoolCommandWrapper.java:40)
at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1687)
at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2016-12-23 11:30:59,111 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-404:ctx-c95759e3) (logid:f96038aa) Seq 1-4225502350380368681: Response Received:
2016-12-23 11:30:59,111 DEBUG [c.c.a.t.Request] (DirectAgent-404:ctx-c95759e3) (logid:f96038aa) Seq 1-4225502350380368681: Processing: { Ans: , MgmtId: 7003044906821, via: 1(pr1854-t722-xenserver-65sp1
-xs1), Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":false,"details":"ModifyStoragePoolCommand add XenAPIException:Unable to setup heartbeat sr on SR ca748d82-3668-9559-07ba-19ff94844ed0
due to #6# Unable to create heartbeat volume hb-db2a4468-ef57-4936-9ccd-7b8f4c06d822 host:db2a4468-ef57-4936-9ccd-7b8f4c06d822 pool: 10.5.0.4/iqn.2016-10.local.marvin:tgt1/0","wait":0}}] }
2016-12-23 11:30:59,111 DEBUG [c.c.a.t.Request] (catalina-exec-18:ctx-ecf415b7 ctx-29dead47 ctx-1fd26575) (logid:f96038aa) Seq 1-4225502350380368681: Received: { Ans: , MgmtId: 7003044906821, via: 1(pr
1854-t722-xenserver-65sp1-xs1), Ver: v1, Flags: 10, { Answer } }
2016-12-23 11:30:59,111 DEBUG [c.c.a.m.AgentManagerImpl] (catalina-exec-18:ctx-ecf415b7 ctx-29dead47 ctx-1fd26575) (logid:f96038aa) Details from executing class com.cloud.agent.api.ModifyStoragePoolComm
and: ModifyStoragePoolCommand add XenAPIException:Unable to setup heartbeat sr on SR ca748d82-3668-9559-07ba-19ff94844ed0 due to #6# Unable to create heartbeat volume hb-db2a4468-ef57-4936-9ccd-7b8f4c06
d822 host:db2a4468-ef57-4936-9ccd-7b8f4c06d822 pool: 10.5.0.4/iqn.2016-10.local.marvin:tgt1/0
2016-12-23 11:30:59,112 WARN [o.a.c.alerts] (catalina-exec-18:ctx-ecf415b7 ctx-29dead47 ctx-1fd26575) (logid:f96038aa) alertType:: 7 // dataCenterId:: 1 // podId:: 1 // clusterId:: null // message:: U
nable to attach storage pool11 to the host1
2016-12-23 11:30:59,120 WARN [o.a.c.s.d.l.CloudStackPrimaryDataStoreLifeCycleImpl] (catalina-exec-18:ctx-ecf415b7 ctx-29dead47 ctx-1fd26575) (logid:f96038aa) Unable to establish a connection between Ho
st[-1-Routing] and org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@70894d68
com.cloud.utils.exception.CloudRuntimeException: Unable establish connection from storage head to storage pool 11 due to ModifyStoragePoolCommand add XenAPIException:Unable to setup heartbeat sr on SR c
a748d82-3668-9559-07ba-19ff94844ed0 due to #6# Unable to create heartbeat volume hb-db2a4468-ef57-4936-9ccd-7b8f4c06d822 host:db2a4468-ef57-4936-9ccd-7b8f4c06d822 pool: 10.5.0.4/iqn.2016-10.local.marvin
:tgt1/011
at org.apache.cloudstack.storage.datastore.provider.DefaultHostListener.hostConnect(DefaultHostListener.java:75)
at com.cloud.storage.StorageManagerImpl.connectHostToSharedPool(StorageManagerImpl.java:955)
at org.apache.cloudstack.storage.datastore.lifecycle.CloudStackPrimaryDataStoreLifeCycleImpl.attachCluster(CloudStackPrimaryDataStoreLifeCycleImpl.java:414)
at com.cloud.storage.StorageManagerImpl.createPool(StorageManagerImpl.java:709)
at com.cloud.storage.StorageManagerImpl.createPool(StorageManagerImpl.java:199)
at org.apache.cloudstack.api.command.admin.storage.CreateStoragePoolCmd.execute(CreateStoragePoolCmd.java:163)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
at com.cloud.api.ApiServer.queueCommand(ApiServer.java:709)
at com.cloud.api.ApiServer.handleRequest(ApiServer.java:533)
at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:299)
at com.cloud.api.ApiServlet$1.run(ApiServlet.java:129)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:126)
at com.cloud.api.ApiServlet.doGet(ApiServlet.java:88)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:723)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:299)
at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889)
at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:732)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2274)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2016-12-23 11:30:59,121 WARN [o.a.c.s.d.l.CloudStackPrimaryDataStoreLifeCycleImpl] (catalina-exec-18:ctx-ecf415b7 ctx-29dead47 ctx-1fd26575) (logid:f96038aa) No host can access storage pool org.apache.
cloudstack.storage.datastore.PrimaryDataStoreImpl@70894d68 on cluster 1
2016-12-23 11:30:59,127 DEBUG [c.c.s.StorageManagerImpl] (catalina-exec-18:ctx-ecf415b7 ctx-29dead47 ctx-1fd26575) (logid:f96038aa) Failed to add data store: Failed to access storage pool
com.cloud.utils.exception.CloudRuntimeException: Failed to access storage pool
at org.apache.cloudstack.storage.datastore.lifecycle.CloudStackPrimaryDataStoreLifeCycleImpl.attachCluster(CloudStackPrimaryDataStoreLifeCycleImpl.java:427)
at com.cloud.storage.StorageManagerImpl.createPool(StorageManagerImpl.java:709)
at com.cloud.storage.StorageManagerImpl.createPool(StorageManagerImpl.java:199)
at org.apache.cloudstack.api.command.admin.storage.CreateStoragePoolCmd.execute(CreateStoragePoolCmd.java:163)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
at com.cloud.api.ApiServer.queueCommand(ApiServer.java:709)
at com.cloud.api.ApiServer.handleRequest(ApiServer.java:533)
at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:299)
at com.cloud.api.ApiServlet$1.run(ApiServlet.java:129)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:126)
at com.cloud.api.ApiServlet.doGet(ApiServlet.java:88)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:723)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:299)
at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889)
at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:732)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2274)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2016-12-23 11:30:59,129 DEBUG [c.c.s.StorageManagerImpl] (catalina-exec-18:ctx-ecf415b7 ctx-29dead47 ctx-1fd26575) (logid:f96038aa) Failed to clean up storage pool: null
2016-12-23 11:30:59,130 INFO [c.c.a.ApiServer] (catalina-exec-18:ctx-ecf415b7 ctx-29dead47 ctx-1fd26575) (logid:f96038aa) Failed to add data store: Failed to access storage pool
2016-12-23 11:30:59,130 DEBUG [c.c.a.ApiServlet] (catalina-exec-18:ctx-ecf415b7 ctx-29dead47 ctx-1fd26575) (logid:f96038aa) ===END=== 10.5.0.4 -- GET apiKey=LIN6rqXuaJwMPfGYFh13qDwYz5VNNz1J2J6qIOWcd3oLQOq0WtD4CwRundBL6rzXToa3lQOC_vKjI3nkHtiD8Q&name=Test+iSCSI&url=iscsi%3A%2F%2F10.5.0.4%2Fiqn.2016-10.local.marvin%3Atgt1%2F0&podid=8b409e63-f298-4113-a23b-0700a8dfc76d&clusterid=4b2458e0-9da9-4a99-987e-3d2cd6f2f0a9&zoneid=8d414040-93d1-47d9-8706-7a15f3c1dfb6&command=createStoragePool&signature=eLjbko60Wfjj4iny2w0ktjvKcK4%3D&response=json
2016-12-23 11:30:59,143 DEBUG [c.c.a.ApiServlet] (catalina-exec-16:ctx-9fd311e8) (logid:2d0d3d40) ===START=== 10.5.0.4 -- GET apiKey=LIN6rqXuaJwMPfGYFh13qDwYz5
Marvin test failure:
2016-12-23 11:30:58,520 - CRITICAL - EXCEPTION: test_01_primary_storage_iscsi: ['Traceback (most recent call last):\n', ' File "/usr/lib64/python2.7/unittest/case.py", line 369, in run\n testMethod()\n', ' File "/marvin/tests/smoke/test_primary_storage.py", line 199, in test_01_primary_storage_iscsi\n podid=self.pod.id\n', ' File "/usr/lib/python2.7/site-packages/marvin/lib/base.py", line 2732, in create\n return StoragePool(apiclient.createStoragePool(cmd).__dict__)\n', ' File "/usr/lib/python2.7/site-packages/marvin/cloudstackAPI/cloudstackAPIClient.py", line 2537, in createStoragePool\n response = self.connection.marvinRequest(command, response_type=response, method=method)\n', ' File "/usr/lib/python2.7/site-packages/marvin/cloudstackConnection.py", line 379, in marvinRequest\n raise e\n', 'CloudstackAPIException: Execute cmd: createstoragepool failed, due to: errorCode: 530, errorText:Failed to add data store: Failed to access storage pool\n']
boris.stoyanov@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue