You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@cloudstack.apache.org by "Sangeetha Hariharan (JIRA)" <ji...@apache.org> on 2013/05/21 01:41:17 UTC
[jira] [Updated] (CLOUDSTACK-2593) Object_Store_Refactor - Not able
to migrate volumes between primary storages with in the same cluster.
[ https://issues.apache.org/jira/browse/CLOUDSTACK-2593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sangeetha Hariharan updated CLOUDSTACK-2593:
--------------------------------------------
Description:
Steps to reproduce the problem:
Set up:
Advanced zone with NFS secondary storage and 1 primary storage ps1 for cluster1.
Deploy a VM.
Add 1 more primary storage ps3 for cluster1.
Stop the Vm.
Migrate volume from ps1 to ps2.
This action reports success.
But the volume did not success fully migrate to ps3.
Logs indicates java.lang.NullPointerException being encountered.
management serve logs:
2013-05-20 15:34:00,197 DEBUG [cloud.api.ApiServlet] (catalina-exec-25:null) ===START=== 10.217.252.128 -- GET command=migrateVirtualMac
hine&storageid=dbf7203e-d359-3179-a464-9900e126cdb0&virtualmachineid=d38d76a2-4a31-484b-9b43-7f11dbc78dde&response=json&sessionkey=Obc59D67KcZYrCEk35NLgMNbhoI%3D&_=1369092666403
2013-05-20 15:34:00,279 DEBUG [cloud.async.AsyncJobManagerImpl] (catalina-exec-25:null) submit async job-18, details: AsyncJobVO {id:18, userId: 2, accountId: 2, sessionKey: null, instanceType: None, instanceId: null, cmd: org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd, cmdOriginator: null, cmdInfo: {"response":"json","sessionkey":"Obc59D67KcZYrCEk35NLgMNbhoI\u003d","virtualmachineid":"d38d76a2-4a31-484b-9b43-7f11dbc78dde","ctxUserId":"2","storageid":"dbf7203e-d359-3179-a464-9900e126cdb0","httpmethod":"GET","_":"1369092666403","ctxAccountId":"2","ctxStartEventId":"62"}, cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0, processStatus: 0, resultCode: 0, result: null, initMsid: 206915885079359, completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
2013-05-20 15:34:00,280 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-18:job-18) Executing org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd for job-18
2013-05-20 15:34:00,282 DEBUG [cloud.api.ApiServlet] (catalina-exec-25:null) ===END=== 10.217.252.128 -- GET command=migrateVirtualMachine&storageid=dbf7203e-d359-3179-a464-9900e126cdb0&virtualmachineid=d38d76a2-4a31-484b-9b43-7f11dbc78dde&response=json&sessionkey=Obc59D67KcZYrCEk35NLgMNbhoI%3D&_=1369092666403
2013-05-20 15:34:00,303 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-18:job-18) VM state transitted from :Stopped to Migrating with event: StorageMigrationRequestedvm's original host id: 1 new host id: null host id before state transition: null
2013-05-20 15:34:00,317 DEBUG [cloud.storage.VolumeManagerImpl] (Job-Executor-18:job-18) migrate volume failed:java.lang.NullPointerException
2013-05-20 15:34:00,317 DEBUG [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-18:job-18) Storage migration failed
2013-05-20 15:34:00,328 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-18:job-18) VM state transitted from :Migrating to Stopped with event: AgentReportStoppedvm's original host id: 1 new host id: null host id before state transition: null
2013-05-20 15:34:00,334 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-18:job-18) Hosts's actual total CPU: 9044 and CPU after applying overprovisioning: 9044
2013-05-20 15:34:00,334 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-18:job-18) Hosts's actual total RAM: 16190149248 and RAM after applying overprovisioning: 16190149632
2013-05-20 15:34:00,334 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-18:job-18) release cpu from host: 1, old used: 1500,reserved: 500, actual total: 9044, total with overprovisioning: 9044; new used: 1000,reserved:500; movedfromreserved: false,moveToReserveredfalse
2013-05-20 15:34:00,334 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-18:job-18) release mem from host: 1, old used: 1476395008,reserved: 536870912, total: 16190149632; new used: 939524096,reserved:536870912; movedfromreserved: false,moveToReserveredfalse
2013-05-20 15:34:00,358 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-18:job-18) Complete async job-18, jobStatus: 1, resultCode: 0, result: org.apache.cloudstack.api.response.UserVmResponse@125e633f
2013-05-20 15:34:00,370 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-18:job-18) Done executing org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd for job-18
2013-05-20 15:34:01,272 DEBUG [cloud.network.NetworkManagerImpl] (Network-Scavenger-1:null) We found network 204 to be free for the first time. Adding it to the list: 1337001212
2013-05-20 15:34:01,633 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-213:null) Ping from 1
2013-05-20 15:34:03,362 DEBUG [cloud.api.ApiServlet] (catalina-exec-20:null) ===START=== 10.217.252.128 -- GET command=queryAsyncJobResult&jobId=35ff84de-f061-4aae-9dac-93fb9cc59f08&response=json&sessionkey=Obc59D67KcZYrCEk35NLgMNbhoI%3D&_=1369092669786
2013-05-20 15:34:03,373 DEBUG [cloud.async.AsyncJobManagerImpl] (catalina-exec-20:null) Async job-18 completed
mysql> select * from storage_pool;
+----+------+--------------------------------------+-------------------+------+----------------+--------+------------+-----------------+----------------+----------------+-----------+-------------------------------------------------+---------------------+---------+-------------+--------+-----------------------+---------+
| id | name | uuid | pool_type | port | data_center_id | pod_id | cluster_id | available_bytes | capacity_bytes | host_address | user_info | path | created | removed | update_time | status | storage_provider_name | scope |
+----+------+--------------------------------------+-------------------+------+----------------+--------+------------+-----------------+----------------+----------------+-----------+-------------------------------------------------+---------------------+---------+-------------+--------+-----------------------+---------+
| 1 | ps1 | e97b5727-d157-3dc1-b32c-065b99806623 | NetworkFilesystem | 2049 | 1 | 1 | 1 | 3018555588608 | 11810778316800 | 10.223.110.232 | NULL | /export/home/sangeetha/campo-systemp-1/primary | 2013-05-20 21:35:08 | NULL | NULL | Up | DefaultPrimary | CLUSTER |
| 2 | ps2 | 3f302dc0-452a-30a0-9115-69c5c1e2b39d | NetworkFilesystem | 2049 | 1 | NULL | NULL | 0 | 0 | 10.223.110.232 | NULL | /export/home/sangeetha/campo-systemp-1/primary1 | 2013-05-20 21:46:10 | NULL | NULL | Up | DefaultPrimary | ZONE |
| 3 | ps3 | dbf7203e-d359-3179-a464-9900e126cdb0 | NetworkFilesystem | 2049 | 1 | 1 | 1 | 3006100733952 | 11810778316800 | 10.223.110.232 | NULL | /export/home/sangeetha/campo-systemp-1/primary2 | 2013-05-20 22:30:03 | NULL | NULL | Up | DefaultPrimary | CLUSTER |
+----+------+--------------------------------------+-------------------+------+----------------+--------+------------+-----------------+----------------+----------------+-----------+-------------------------------------------------+---------------------+---------+-------------+--------+-----------------------+---------+
3 rows in set (0.00 sec)
was:
management serve logs:
2013-05-20 15:34:00,197 DEBUG [cloud.api.ApiServlet] (catalina-exec-25:null) ===START=== 10.217.252.128 -- GET command=migrateVirtualMac
hine&storageid=dbf7203e-d359-3179-a464-9900e126cdb0&virtualmachineid=d38d76a2-4a31-484b-9b43-7f11dbc78dde&response=json&sessionkey=Obc59D67KcZYrCEk35NLgMNbhoI%3D&_=1369092666403
2013-05-20 15:34:00,279 DEBUG [cloud.async.AsyncJobManagerImpl] (catalina-exec-25:null) submit async job-18, details: AsyncJobVO {id:18, userId: 2, accountId: 2, sessionKey: null, instanceType: None, instanceId: null, cmd: org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd, cmdOriginator: null, cmdInfo: {"response":"json","sessionkey":"Obc59D67KcZYrCEk35NLgMNbhoI\u003d","virtualmachineid":"d38d76a2-4a31-484b-9b43-7f11dbc78dde","ctxUserId":"2","storageid":"dbf7203e-d359-3179-a464-9900e126cdb0","httpmethod":"GET","_":"1369092666403","ctxAccountId":"2","ctxStartEventId":"62"}, cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0, processStatus: 0, resultCode: 0, result: null, initMsid: 206915885079359, completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
2013-05-20 15:34:00,280 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-18:job-18) Executing org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd for job-18
2013-05-20 15:34:00,282 DEBUG [cloud.api.ApiServlet] (catalina-exec-25:null) ===END=== 10.217.252.128 -- GET command=migrateVirtualMachine&storageid=dbf7203e-d359-3179-a464-9900e126cdb0&virtualmachineid=d38d76a2-4a31-484b-9b43-7f11dbc78dde&response=json&sessionkey=Obc59D67KcZYrCEk35NLgMNbhoI%3D&_=1369092666403
2013-05-20 15:34:00,303 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-18:job-18) VM state transitted from :Stopped to Migrating with event: StorageMigrationRequestedvm's original host id: 1 new host id: null host id before state transition: null
2013-05-20 15:34:00,317 DEBUG [cloud.storage.VolumeManagerImpl] (Job-Executor-18:job-18) migrate volume failed:java.lang.NullPointerException
2013-05-20 15:34:00,317 DEBUG [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-18:job-18) Storage migration failed
2013-05-20 15:34:00,328 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-18:job-18) VM state transitted from :Migrating to Stopped with event: AgentReportStoppedvm's original host id: 1 new host id: null host id before state transition: null
2013-05-20 15:34:00,334 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-18:job-18) Hosts's actual total CPU: 9044 and CPU after applying overprovisioning: 9044
2013-05-20 15:34:00,334 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-18:job-18) Hosts's actual total RAM: 16190149248 and RAM after applying overprovisioning: 16190149632
2013-05-20 15:34:00,334 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-18:job-18) release cpu from host: 1, old used: 1500,reserved: 500, actual total: 9044, total with overprovisioning: 9044; new used: 1000,reserved:500; movedfromreserved: false,moveToReserveredfalse
2013-05-20 15:34:00,334 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-18:job-18) release mem from host: 1, old used: 1476395008,reserved: 536870912, total: 16190149632; new used: 939524096,reserved:536870912; movedfromreserved: false,moveToReserveredfalse
2013-05-20 15:34:00,358 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-18:job-18) Complete async job-18, jobStatus: 1, resultCode: 0, result: org.apache.cloudstack.api.response.UserVmResponse@125e633f
2013-05-20 15:34:00,370 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-18:job-18) Done executing org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd for job-18
2013-05-20 15:34:01,272 DEBUG [cloud.network.NetworkManagerImpl] (Network-Scavenger-1:null) We found network 204 to be free for the first time. Adding it to the list: 1337001212
2013-05-20 15:34:01,633 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-213:null) Ping from 1
2013-05-20 15:34:03,362 DEBUG [cloud.api.ApiServlet] (catalina-exec-20:null) ===START=== 10.217.252.128 -- GET command=queryAsyncJobResult&jobId=35ff84de-f061-4aae-9dac-93fb9cc59f08&response=json&sessionkey=Obc59D67KcZYrCEk35NLgMNbhoI%3D&_=1369092669786
2013-05-20 15:34:03,373 DEBUG [cloud.async.AsyncJobManagerImpl] (catalina-exec-20:null) Async job-18 completed
mysql> select * from storage_pool;
+----+------+--------------------------------------+-------------------+------+----------------+--------+------------+-----------------+----------------+----------------+-----------+-------------------------------------------------+---------------------+---------+-------------+--------+-----------------------+---------+
| id | name | uuid | pool_type | port | data_center_id | pod_id | cluster_id | available_bytes | capacity_bytes | host_address | user_info | path | created | removed | update_time | status | storage_provider_name | scope |
+----+------+--------------------------------------+-------------------+------+----------------+--------+------------+-----------------+----------------+----------------+-----------+-------------------------------------------------+---------------------+---------+-------------+--------+-----------------------+---------+
| 1 | ps1 | e97b5727-d157-3dc1-b32c-065b99806623 | NetworkFilesystem | 2049 | 1 | 1 | 1 | 3018555588608 | 11810778316800 | 10.223.110.232 | NULL | /export/home/sangeetha/campo-systemp-1/primary | 2013-05-20 21:35:08 | NULL | NULL | Up | DefaultPrimary | CLUSTER |
| 2 | ps2 | 3f302dc0-452a-30a0-9115-69c5c1e2b39d | NetworkFilesystem | 2049 | 1 | NULL | NULL | 0 | 0 | 10.223.110.232 | NULL | /export/home/sangeetha/campo-systemp-1/primary1 | 2013-05-20 21:46:10 | NULL | NULL | Up | DefaultPrimary | ZONE |
| 3 | ps3 | dbf7203e-d359-3179-a464-9900e126cdb0 | NetworkFilesystem | 2049 | 1 | 1 | 1 | 3006100733952 | 11810778316800 | 10.223.110.232 | NULL | /export/home/sangeetha/campo-systemp-1/primary2 | 2013-05-20 22:30:03 | NULL | NULL | Up | DefaultPrimary | CLUSTER |
+----+------+--------------------------------------+-------------------+------+----------------+--------+------------+-----------------+----------------+----------------+-----------+-------------------------------------------------+---------------------+---------+-------------+--------+-----------------------+---------+
3 rows in set (0.00 sec)
> Object_Store_Refactor - Not able to migrate volumes between primary storages with in the same cluster.
> ------------------------------------------------------------------------------------------------------
>
> Key: CLOUDSTACK-2593
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-2593
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the default.)
> Affects Versions: 4.2.0
> Environment: Build from object_store
> Reporter: Sangeetha Hariharan
> Priority: Critical
> Fix For: 4.2.0
>
>
> Steps to reproduce the problem:
> Set up:
> Advanced zone with NFS secondary storage and 1 primary storage ps1 for cluster1.
> Deploy a VM.
> Add 1 more primary storage ps3 for cluster1.
> Stop the Vm.
> Migrate volume from ps1 to ps2.
> This action reports success.
> But the volume did not success fully migrate to ps3.
> Logs indicates java.lang.NullPointerException being encountered.
> management serve logs:
> 2013-05-20 15:34:00,197 DEBUG [cloud.api.ApiServlet] (catalina-exec-25:null) ===START=== 10.217.252.128 -- GET command=migrateVirtualMac
> hine&storageid=dbf7203e-d359-3179-a464-9900e126cdb0&virtualmachineid=d38d76a2-4a31-484b-9b43-7f11dbc78dde&response=json&sessionkey=Obc59D67KcZYrCEk35NLgMNbhoI%3D&_=1369092666403
> 2013-05-20 15:34:00,279 DEBUG [cloud.async.AsyncJobManagerImpl] (catalina-exec-25:null) submit async job-18, details: AsyncJobVO {id:18, userId: 2, accountId: 2, sessionKey: null, instanceType: None, instanceId: null, cmd: org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd, cmdOriginator: null, cmdInfo: {"response":"json","sessionkey":"Obc59D67KcZYrCEk35NLgMNbhoI\u003d","virtualmachineid":"d38d76a2-4a31-484b-9b43-7f11dbc78dde","ctxUserId":"2","storageid":"dbf7203e-d359-3179-a464-9900e126cdb0","httpmethod":"GET","_":"1369092666403","ctxAccountId":"2","ctxStartEventId":"62"}, cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0, processStatus: 0, resultCode: 0, result: null, initMsid: 206915885079359, completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
> 2013-05-20 15:34:00,280 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-18:job-18) Executing org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd for job-18
> 2013-05-20 15:34:00,282 DEBUG [cloud.api.ApiServlet] (catalina-exec-25:null) ===END=== 10.217.252.128 -- GET command=migrateVirtualMachine&storageid=dbf7203e-d359-3179-a464-9900e126cdb0&virtualmachineid=d38d76a2-4a31-484b-9b43-7f11dbc78dde&response=json&sessionkey=Obc59D67KcZYrCEk35NLgMNbhoI%3D&_=1369092666403
> 2013-05-20 15:34:00,303 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-18:job-18) VM state transitted from :Stopped to Migrating with event: StorageMigrationRequestedvm's original host id: 1 new host id: null host id before state transition: null
> 2013-05-20 15:34:00,317 DEBUG [cloud.storage.VolumeManagerImpl] (Job-Executor-18:job-18) migrate volume failed:java.lang.NullPointerException
> 2013-05-20 15:34:00,317 DEBUG [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-18:job-18) Storage migration failed
> 2013-05-20 15:34:00,328 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-18:job-18) VM state transitted from :Migrating to Stopped with event: AgentReportStoppedvm's original host id: 1 new host id: null host id before state transition: null
> 2013-05-20 15:34:00,334 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-18:job-18) Hosts's actual total CPU: 9044 and CPU after applying overprovisioning: 9044
> 2013-05-20 15:34:00,334 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-18:job-18) Hosts's actual total RAM: 16190149248 and RAM after applying overprovisioning: 16190149632
> 2013-05-20 15:34:00,334 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-18:job-18) release cpu from host: 1, old used: 1500,reserved: 500, actual total: 9044, total with overprovisioning: 9044; new used: 1000,reserved:500; movedfromreserved: false,moveToReserveredfalse
> 2013-05-20 15:34:00,334 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-18:job-18) release mem from host: 1, old used: 1476395008,reserved: 536870912, total: 16190149632; new used: 939524096,reserved:536870912; movedfromreserved: false,moveToReserveredfalse
> 2013-05-20 15:34:00,358 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-18:job-18) Complete async job-18, jobStatus: 1, resultCode: 0, result: org.apache.cloudstack.api.response.UserVmResponse@125e633f
> 2013-05-20 15:34:00,370 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-18:job-18) Done executing org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd for job-18
> 2013-05-20 15:34:01,272 DEBUG [cloud.network.NetworkManagerImpl] (Network-Scavenger-1:null) We found network 204 to be free for the first time. Adding it to the list: 1337001212
> 2013-05-20 15:34:01,633 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-213:null) Ping from 1
> 2013-05-20 15:34:03,362 DEBUG [cloud.api.ApiServlet] (catalina-exec-20:null) ===START=== 10.217.252.128 -- GET command=queryAsyncJobResult&jobId=35ff84de-f061-4aae-9dac-93fb9cc59f08&response=json&sessionkey=Obc59D67KcZYrCEk35NLgMNbhoI%3D&_=1369092669786
> 2013-05-20 15:34:03,373 DEBUG [cloud.async.AsyncJobManagerImpl] (catalina-exec-20:null) Async job-18 completed
> mysql> select * from storage_pool;
> +----+------+--------------------------------------+-------------------+------+----------------+--------+------------+-----------------+----------------+----------------+-----------+-------------------------------------------------+---------------------+---------+-------------+--------+-----------------------+---------+
> | id | name | uuid | pool_type | port | data_center_id | pod_id | cluster_id | available_bytes | capacity_bytes | host_address | user_info | path | created | removed | update_time | status | storage_provider_name | scope |
> +----+------+--------------------------------------+-------------------+------+----------------+--------+------------+-----------------+----------------+----------------+-----------+-------------------------------------------------+---------------------+---------+-------------+--------+-----------------------+---------+
> | 1 | ps1 | e97b5727-d157-3dc1-b32c-065b99806623 | NetworkFilesystem | 2049 | 1 | 1 | 1 | 3018555588608 | 11810778316800 | 10.223.110.232 | NULL | /export/home/sangeetha/campo-systemp-1/primary | 2013-05-20 21:35:08 | NULL | NULL | Up | DefaultPrimary | CLUSTER |
> | 2 | ps2 | 3f302dc0-452a-30a0-9115-69c5c1e2b39d | NetworkFilesystem | 2049 | 1 | NULL | NULL | 0 | 0 | 10.223.110.232 | NULL | /export/home/sangeetha/campo-systemp-1/primary1 | 2013-05-20 21:46:10 | NULL | NULL | Up | DefaultPrimary | ZONE |
> | 3 | ps3 | dbf7203e-d359-3179-a464-9900e126cdb0 | NetworkFilesystem | 2049 | 1 | 1 | 1 | 3006100733952 | 11810778316800 | 10.223.110.232 | NULL | /export/home/sangeetha/campo-systemp-1/primary2 | 2013-05-20 22:30:03 | NULL | NULL | Up | DefaultPrimary | CLUSTER |
> +----+------+--------------------------------------+-------------------+------+----------------+--------+------------+-----------------+----------------+----------------+-----------+-------------------------------------------------+---------------------+---------+-------------+--------+-----------------------+---------+
> 3 rows in set (0.00 sec)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira