You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by he...@powershield.ru on 2017/11/06 11:14:21 UTC
VR starting problem
Hello,
ACS: 4.8.0
XenServer 6.5SP1
We have got some problem with additional space for our client.
We are created new Disk Offering, then we tryed to attach for our
client, but got error: Unable to find suitable primary storage when
creating volume
We checked datadisk and we see, there is not hypervisor (should it be
showed? Now saying: None)
We tryed to create new instance, but also unlucky.
After it, we restarted VR and he can't start. He started and again goes
to offline. We tryed to check from XenCenter and got this reason:
apache2: Could not reliably determine the server's fully qualified
domain name, using 127.0.0.1 for ServerName.
We delete VR and now, we can not start it manually and he is not
starting automatically about 40min.
Re: VR starting problem
Posted by he...@powershield.ru.
Insufficient space on pool: 6 since its usage percentage:
0.8669012505498059 has crossed the
pool.storage.capacity.disablethreshold: 0.85
Checked now, and i see, what storage capacity is only 70%
Ivan Kudryavtsev писал 2017-11-06 19:51:
> Hi. Try to find in early logs what exactly resource is insufficient.
> Try to
> play with threshold of it.
>
> 6 нояб. 2017 г. 21:57 пользователь <he...@powershield.ru> написал:
>
>> Now, i see the problem, then VR restarting.
>>
>> 2017-11-06 11:54:15,821 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-43:ctx-02214ed9 job-14455) (logid:18ad84ad) Complete
>> async job-14455, jobStatus: FAILED, resultCode: 530, result:
>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>> uuidList":[],"errorcode":530,"errortext":"Resource [Host:17] is
>> unreachable: Host 17: Unable to start instance due to Unable to start
>> VM:18985a14-3cb5-46d5-a3f0-3a065cd8b014 due to error in finalizeStart,
>> not retrying"}
>> 2017-11-06 15:47:27,814 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-69:ctx-be0b5550 job-14663) (logid:8b609324) Complete
>> async job-14663, jobStatus: FAILED, resultCode: 530, result:
>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>> uuidList":[],"errorcode":530,"errortext":"Unable to start a VM due to
>> insufficient capacity"}
>> 2017-11-06 16:54:19,055 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-70:ctx-deaaf8e8 job-14668) (logid:6a7700af) Complete
>> async job-14668, jobStatus: FAILED, resultCode: 530, result:
>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>> uuidList":[],"errorcode":530,"errortext":"Resource [DataCenter:2] is
>> unreachable: Unable to reboot domR, it is not in right state Stopped"}
>> 2017-11-06 16:58:54,235 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-71:ctx-b26b3b33 job-14669) (logid:436e6183) Complete
>> async job-14669, jobStatus: FAILED, resultCode: 530, result:
>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>> uuidList":[],"errorcode":530,"errortext":"Unable to create a
>> deployment
>> for VM[DomainRouter|r-1919-VM]"}
>>
>> I'm pretty sure, what i have enough resources:
>>
>> Primary Storage Used
>> Allocated: 4.22 TB / 4.88 TB
>>
>> Primary Storage Allocated
>> Allocated: 3.02 TB / 4.88 TB
>>
>> Memory Allocated
>> Allocated: 78.00 GB / 597.93 GB
>>
>> CPU Allocated
>> Allocated: 81.00 GHz / 440.00 GHz
>>
>> Shared Network IPs
>> Allocated: 34 / 253
>>
>>
>>
>>
>> Ivan Kudryavtsev писал 2017-11-06 15:28:
>>
>>> Hi, you should dive into management server logs to find relevant
>>> messages.
>>> Otherwise it's impossible to predict what's wrong with the system.
>>> Check
>>> logs and find exceptions and other errornous stuff.
>>>
>>> 6 нояб. 2017 г. 18:14 пользователь <he...@powershield.ru> написал:
>>>
>>> Hello,
>>>>
>>>> ACS: 4.8.0
>>>> XenServer 6.5SP1
>>>>
>>>>
>>>> We have got some problem with additional space for our client.
>>>>
>>>> We are created new Disk Offering, then we tryed to attach for our
>>>> client,
>>>> but got error: Unable to find suitable primary storage when creating
>>>> volume
>>>>
>>>> We checked datadisk and we see, there is not hypervisor (should it
>>>> be
>>>> showed? Now saying: None)
>>>>
>>>> We tryed to create new instance, but also unlucky.
>>>>
>>>> After it, we restarted VR and he can't start. He started and again
>>>> goes
>>>> to
>>>> offline. We tryed to check from XenCenter and got this reason:
>>>> apache2:
>>>> Could not reliably determine the server's fully qualified domain
>>>> name,
>>>> using 127.0.0.1 for ServerName.
>>>>
>>>> We delete VR and now, we can not start it manually and he is not
>>>> starting
>>>> automatically about 40min.
>>>>
>>>>
Re: VR starting problem
Posted by he...@powershield.ru.
Checking pool 6 for storage, totalSize: 5497545555968, usedBytes:
3634788040704, usedPct: 0.6611656063055564, disable threshold: 0.98
Checking pool: 6 for volume allocation [Vol[2426|vm=2402|ROOT]], maxSize
: 5497545555968, totalAllocatedSize : 3328936879616, askingSize :
3151880704, allocated disable threshold: 0.98
https://pastebin.com/A5Z4zA2U
Paul Angus писал 2017-11-07 00:18:
> Yes, I see that, but as the message is still the same, you either
> changed a similar but different setting (there are a number of
> similarly named global settings) or you did not restart all of your
> management servers.
>
>
>
> Kind regards,
>
> Paul Angus
>
> paul.angus@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> @shapeblue
>
>
>
>
> -----Original Message-----
> From: hello@powershield.ru [mailto:hello@powershield.ru]
> Sent: 06 November 2017 21:15
> To: users@cloudstack.apache.org
> Subject: Re: VR starting problem
>
> As i wrote before, i already did it before first my e-mail here, but
> without luck:)
>
>
>
> Paul Angus писал 2017-11-07 00:12:
>> Ideally we'd need to start the log even earlier, but I can still see:
>> Insufficient space on pool: 6 since its usage percentage:
>> 0.8669012505498059 has crossed the
>> pool.storage.capacity.disablethreshold: 0.85
>>
>> If you increased pool.storage.capacity.disablethreshold to above 087,
>> then you need to restart your management server(s) for it to take
>> effect.
>>
>> It would be better though, if you add another primary storage pool, or
>> increase the storage in the existing pool. As increasing the threshold
>> isn't a long term solution.
>>
>>
>>
>>
>>
>>
>> Kind regards,
>>
>> Paul Angus
>>
>> paul.angus@shapeblue.com
>> www.shapeblue.com
>> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
>> @shapeblue
>>
>>
>>
>>
>> -----Original Message-----
>> From: hello@powershield.ru [mailto:hello@powershield.ru]
>> Sent: 06 November 2017 20:53
>> To: users@cloudstack.apache.org
>> Subject: Re: VR starting problem
>>
>> Hello,
>>
>> Here is some of log.
>>
>> https://pastebin.com/JU0qr69V
>>
>> Interested line: dataCenterId:: 2 // podId:: null // clusterId:: null
>>
>> As you can see, datacenter (zone) showing, but can't see pod and
>> cluster....
>>
>>
>> Paul Angus писал 2017-11-06 22:44:
>>> Please could you use something like pastebin (https://pastebin.com/)
>>> to paste larger sections of your logs.
>>> The section you have pasted is more likely from XenServer trying to
>>> remove a disk image that was never created in the first place. It
>>> may
>>> not be the root cause.
>>>
>>> Kind regards,
>>>
>>> Paul Angus
>>>
>>> paul.angus@shapeblue.com
>>> www.shapeblue.com
>>> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
>>> @shapeblue
>>>
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: hello@powershield.ru [mailto:hello@powershield.ru]
>>> Sent: 06 November 2017 18:14
>>> To: users@cloudstack.apache.org
>>> Subject: Re: VR starting problem
>>>
>>> We deleted system VM's and also, can't start.
>>>
>>> Now seeing the error:
>>>
>>>
>>> INFO [c.c.v.VirtualMachineManagerImpl]
>>> (Work-Job-Executor-11:ctx-eaf4ab20 job-97/job-14993 ctx-7391ff7c)
>>> (logid:66199d2c) Unable to contact resource.
>>> com.cloud.exception.StorageUnavailableException: Resource
>>> [StoragePool:6] is unreachable: Unable to create
>>> Vol[2034|vm=2010|ROOT]:com.cloud.utils.exception.CloudRuntimeException:
>>> Catch Exception com.xensource.xenapi.Types$UuidInvalid :VDI getByUuid
>>> for uuid: 5e2f645f-9e5d-45df-a268-dbe9c6c9eef9 failed due to The uuid
>>> you supplied was invalid.
>>>
>>> 2017-11-06 20:16:49,498 DEBUG [c.c.h.x.r.XenServerStorageProcessor]
>>> (DirectAgent-6:ctx-b7ab797e) (logid:d3084120) Catch Exception
>>> com.xensource.xenapi.Types$UuidInvalid :VDI getByUuid for uuid:
>>> 5e2f645f-9e5d-45df-a268-dbe9c6c9eef9 failed due to The uuid you
>>> supplied was invalid.
>>> 2017-11-06 20:16:49,501 WARN [c.c.h.x.r.XenServerStorageProcessor]
>>> (DirectAgent-6:ctx-b7ab797e) (logid:d3084120) Unable to create
>>> volume;
>>> Pool=volumeTO[uuid=efa30838-f8e4-4186-a299-d2ae716d9081|path=null|datastore=PrimaryDataStoreTO[uuid=SR-UK|name=null|id=6|pooltype=PreSetup]];
>>> Disk:
>>> com.cloud.utils.exception.CloudRuntimeException: Catch Exception
>>> com.xensource.xenapi.Types$UuidInvalid :VDI getByUuid for uuid:
>>> 5e2f645f-9e5d-45df-a268-dbe9c6c9eef9 failed due to The uuid you
>>> supplied was invalid.
>>>
>>>
>>>
>>>
>>>
>>> hello@powershield.ru писал 2017-11-06 20:48:
>>>> Changed few hours ago, and anyway still the same problem.
>>>>
>>>>
>>>>
>>>> Ivan Kudryavtsev писал 2017-11-06 20:40:
>>>>> Hi. Don't mix "used" and "allocated". Just change the specified
>>>>> threshold to 0.98 and go ahead.
>>>>>
>>>>> 7 нояб. 2017 г. 0:22 пользователь <he...@powershield.ru> написал:
>>>>>
>>>>>> No, i re-checked maybe 10 times. I just added some more space, but
>>>>>> the problem is same. All VM's working but if client restart - they
>>>>>> goes down, with the same error with insufficient storage.
>>>>>>
>>>>>>
>>>>>> Ivan Kudryavtsev писал 2017-11-06 20:16:
>>>>>>
>>>>>>> Could it be, that your storage, cluster, pod, zone is in the
>>>>>>> maintanence mode?
>>>>>>>
>>>>>>> 6 нояб. 2017 г. 23:51 пользователь "Ivan Kudryavtsev" <
>>>>>>> kudryavtsev_ia@bw-sw.com> написал:
>>>>>>>
>>>>>>> Hi. Try to find in early logs what exactly resource is
>>>>>>> insufficient.
>>>>>>> Try
>>>>>>>> to play with threshold of it.
>>>>>>>>
>>>>>>>> 6 нояб. 2017 г. 21:57 пользователь <he...@powershield.ru>
>>>>>>>> написал:
>>>>>>>>
>>>>>>>> Now, i see the problem, then VR restarting.
>>>>>>>>>
>>>>>>>>> 2017-11-06 11:54:15,821 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>>>> (API-Job-Executor-43:ctx-02214ed9 job-14455) (logid:18ad84ad)
>>>>>>>>> Complete async job-14455, jobStatus: FAILED, resultCode: 530,
>>>>>>>>> result:
>>>>>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>>>>>>>>> uuidList":[],"errorcode":530,"errortext":"Resource [Host:17] is
>>>>>>>>> unreachable: Host 17: Unable to start instance due to Unable to
>>>>>>>>> start
>>>>>>>>> VM:18985a14-3cb5-46d5-a3f0-3a065cd8b014 due to error in
>>>>>>>>> finalizeStart, not retrying"}
>>>>>>>>> 2017-11-06 15:47:27,814 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>>>> (API-Job-Executor-69:ctx-be0b5550 job-14663) (logid:8b609324)
>>>>>>>>> Complete async job-14663, jobStatus: FAILED, resultCode: 530,
>>>>>>>>> result:
>>>>>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>>>>>>>>> uuidList":[],"errorcode":530,"errortext":"Unable to start a VM
>>>>>>>>> due to insufficient capacity"}
>>>>>>>>> 2017-11-06 16:54:19,055 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>>>> (API-Job-Executor-70:ctx-deaaf8e8 job-14668) (logid:6a7700af)
>>>>>>>>> Complete async job-14668, jobStatus: FAILED, resultCode: 530,
>>>>>>>>> result:
>>>>>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>>>>>>>>> uuidList":[],"errorcode":530,"errortext":"Resource
>>>>>>>>> [DataCenter:2]
>>>>>>>>> is
>>>>>>>>> unreachable: Unable to reboot domR, it is not in right state
>>>>>>>>> Stopped"}
>>>>>>>>> 2017-11-06 16:58:54,235 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>>>> (API-Job-Executor-71:ctx-b26b3b33 job-14669) (logid:436e6183)
>>>>>>>>> Complete async job-14669, jobStatus: FAILED, resultCode: 530,
>>>>>>>>> result:
>>>>>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>>>>>>>>> uuidList":[],"errorcode":530,"errortext":"Unable to create a
>>>>>>>>> deployment for VM[DomainRouter|r-1919-VM]"}
>>>>>>>>>
>>>>>>>>> I'm pretty sure, what i have enough resources:
>>>>>>>>>
>>>>>>>>> Primary Storage Used
>>>>>>>>> Allocated: 4.22 TB / 4.88 TB
>>>>>>>>>
>>>>>>>>> Primary Storage Allocated
>>>>>>>>> Allocated: 3.02 TB / 4.88 TB
>>>>>>>>>
>>>>>>>>> Memory Allocated
>>>>>>>>> Allocated: 78.00 GB / 597.93 GB
>>>>>>>>>
>>>>>>>>> CPU Allocated
>>>>>>>>> Allocated: 81.00 GHz / 440.00 GHz
>>>>>>>>>
>>>>>>>>> Shared Network IPs
>>>>>>>>> Allocated: 34 / 253
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Ivan Kudryavtsev писал 2017-11-06 15:28:
>>>>>>>>>
>>>>>>>>> Hi, you should dive into management server logs to find
>>>>>>>>> relevant
>>>>>>>>>> messages.
>>>>>>>>>> Otherwise it's impossible to predict what's wrong with the
>>>>>>>>>> system.
>>>>>>>>>> Check
>>>>>>>>>> logs and find exceptions and other errornous stuff.
>>>>>>>>>>
>>>>>>>>>> 6 нояб. 2017 г. 18:14 пользователь <he...@powershield.ru>
>>>>>>>>>> написал:
>>>>>>>>>>
>>>>>>>>>> Hello,
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ACS: 4.8.0
>>>>>>>>>>> XenServer 6.5SP1
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> We have got some problem with additional space for our
>>>>>>>>>>> client.
>>>>>>>>>>>
>>>>>>>>>>> We are created new Disk Offering, then we tryed to attach for
>>>>>>>>>>> our client, but got error: Unable to find suitable primary
>>>>>>>>>>> storage when creating volume
>>>>>>>>>>>
>>>>>>>>>>> We checked datadisk and we see, there is not hypervisor
>>>>>>>>>>> (should
>>>>>>>>>>> it be showed? Now saying: None)
>>>>>>>>>>>
>>>>>>>>>>> We tryed to create new instance, but also unlucky.
>>>>>>>>>>>
>>>>>>>>>>> After it, we restarted VR and he can't start. He started and
>>>>>>>>>>> again goes to offline. We tryed to check from XenCenter and
>>>>>>>>>>> got
>>>>>>>>>>> this reason:
>>>>>>>>>>> apache2:
>>>>>>>>>>> Could not reliably determine the server's fully qualified
>>>>>>>>>>> domain name, using 127.0.0.1 for ServerName.
>>>>>>>>>>>
>>>>>>>>>>> We delete VR and now, we can not start it manually and he is
>>>>>>>>>>> not starting automatically about 40min.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
Re: Creating a CloudStack advance networking
Posted by Boris Stoyanov <bo...@shapeblue.com>.
Hi Luis,
Can you pease tell us what error do you get on your download.
Could you please describe the exact scenario you’re executing? Are you downloading an ISO from ACS or you’ve registered ISO and the download fails?
First place to start looking is the management logs sitting at /var/log/cloudstack/management/
Thanks,
Bobby.
boris.stoyanov@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
> On 9 Nov 2017, at 23:03, Luis <lm...@yahoo.com.INVALID> wrote:
>
> I am i stalling a demo environment to test CS Advance networking but i am ha ing some problems, i hace CS installed and configured with Advance Networking but is not able to download iso. I am using CS 4.10 with XenServer with loca IPs because is for internal use and to learn.
> Could some o e give me advice or guide me to finish this project?
>
> Firewall pfsense with 3 i terfacesPublic:192.168.0.0/24Servers:10.0.0.0/23Lan:192.168.1.0/23
> Host: 10.0.0.0/23Publuc-local IPs: 192.168.0.0/24
>
>
> Thank you.
Creating a CloudStack advance networking
Posted by Luis <lm...@yahoo.com.INVALID>.
I am i stalling a demo environment to test CS Advance networking but i am ha ing some problems, i hace CS installed and configured with Advance Networking but is not able to download iso. I am using CS 4.10 with XenServer with loca IPs because is for internal use and to learn.
Could some o e give me advice or guide me to finish this project?
Firewall pfsense with 3 i terfacesPublic:192.168.0.0/24Servers:10.0.0.0/23Lan:192.168.1.0/23
Host: 10.0.0.0/23Publuc-local IPs: 192.168.0.0/24
Thank you.
Re: VR starting problem
Posted by he...@powershield.ru.
Спасибо Сергей:)
Thank you, but actually fix was very easy (2 days i search
it....)....Just removed entries from database template_spool_ref
Sergey Levitskiy писал 2017-11-08 03:07:
> Try one of these:
> https://discussions.citrix.com/topic/299938-error-when-disabling-ha-the-uuid-you-supplied-was-invalid/
> http://blog.branix.net/cloudplatform/when-live-storage-migration-goes-wrong
>
>
> On 11/7/17, 12:58 PM, "hello@powershield.ru" <he...@powershield.ru>
> wrote:
>
> All day we tested and have no idea where is a problem:)
>
> 1. We can create template from stopped VM (after stopping VM, we
> can't
> start with the same error: Insufficient capacity)
> 2. We can download and install ISO
> 3. We can create new DATADISK and attach it to online VM's
> 4. We can create new Disk Offering. Create new DATADISK and
> attach/detach from the VM's.
>
> We try'ed to search VDI, because some error's with the same VDI
> uuid:
> c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-1:ctx-bea5904c)
> (logid:1f164ea4) Catch Exception
> com.xensource.xenapi.Types$UuidInvalid
> :VDI getByUuid for uuid: 5e2f645f-9e5d-45df-a268-dbe9c6c9eef9
> failed due
> to The uuid you supplied was invalid.
>
> Where in database writed this uuid? We thinking, here could be a
> problem. We can not find any entries by this uuid in our storage.
>
>
> com.cloud.hypervisor.xenserver.resource.XenServerStorageProcessor.getVDIbyUuid(XenServerStorageProcessor.java:743)
> Caused by: The uuid you supplied was invalid.
>
> Unsupported data object (VOLUME,
>
> org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@4ddd9112),
> no need to delete from object in store ref table
> com.cloud.exception.StorageUnavailableException: Resource
> [StoragePool:6] is unreachable: Unable to create
>
> Vol[5128|vm=5101|ROOT]:com.cloud.utils.exception.CloudRuntimeException:
> Catch Exception com.xensource.xenapi.Types$UuidInvalid :VDI
> getByUuid
> for uuid: 5e2f645f-9e5d-45df-a268-dbe9c6c9eef9 failed due to The
> uuid
> you supplied was invalid.
>
> It's also writing, that storage is unreachable Resource
> [StoragePool:6]
> is unreachable: Unable to create
>
> But as i wrote, we can create new diskdata and so on on this
> storage
> without problem.
>
>
> https://pastebin.com/HLt40fK0
Re: VR starting problem
Posted by Sergey Levitskiy <se...@hotmail.com>.
Try one of these:
https://discussions.citrix.com/topic/299938-error-when-disabling-ha-the-uuid-you-supplied-was-invalid/
http://blog.branix.net/cloudplatform/when-live-storage-migration-goes-wrong
On 11/7/17, 12:58 PM, "hello@powershield.ru" <he...@powershield.ru> wrote:
All day we tested and have no idea where is a problem:)
1. We can create template from stopped VM (after stopping VM, we can't
start with the same error: Insufficient capacity)
2. We can download and install ISO
3. We can create new DATADISK and attach it to online VM's
4. We can create new Disk Offering. Create new DATADISK and
attach/detach from the VM's.
We try'ed to search VDI, because some error's with the same VDI uuid:
c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-1:ctx-bea5904c)
(logid:1f164ea4) Catch Exception com.xensource.xenapi.Types$UuidInvalid
:VDI getByUuid for uuid: 5e2f645f-9e5d-45df-a268-dbe9c6c9eef9 failed due
to The uuid you supplied was invalid.
Where in database writed this uuid? We thinking, here could be a
problem. We can not find any entries by this uuid in our storage.
com.cloud.hypervisor.xenserver.resource.XenServerStorageProcessor.getVDIbyUuid(XenServerStorageProcessor.java:743)
Caused by: The uuid you supplied was invalid.
Unsupported data object (VOLUME,
org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@4ddd9112),
no need to delete from object in store ref table
com.cloud.exception.StorageUnavailableException: Resource
[StoragePool:6] is unreachable: Unable to create
Vol[5128|vm=5101|ROOT]:com.cloud.utils.exception.CloudRuntimeException:
Catch Exception com.xensource.xenapi.Types$UuidInvalid :VDI getByUuid
for uuid: 5e2f645f-9e5d-45df-a268-dbe9c6c9eef9 failed due to The uuid
you supplied was invalid.
It's also writing, that storage is unreachable Resource [StoragePool:6]
is unreachable: Unable to create
But as i wrote, we can create new diskdata and so on on this storage
without problem.
https://pastebin.com/HLt40fK0
Re: VR starting problem
Posted by he...@powershield.ru.
All day we tested and have no idea where is a problem:)
1. We can create template from stopped VM (after stopping VM, we can't
start with the same error: Insufficient capacity)
2. We can download and install ISO
3. We can create new DATADISK and attach it to online VM's
4. We can create new Disk Offering. Create new DATADISK and
attach/detach from the VM's.
We try'ed to search VDI, because some error's with the same VDI uuid:
c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-1:ctx-bea5904c)
(logid:1f164ea4) Catch Exception com.xensource.xenapi.Types$UuidInvalid
:VDI getByUuid for uuid: 5e2f645f-9e5d-45df-a268-dbe9c6c9eef9 failed due
to The uuid you supplied was invalid.
Where in database writed this uuid? We thinking, here could be a
problem. We can not find any entries by this uuid in our storage.
com.cloud.hypervisor.xenserver.resource.XenServerStorageProcessor.getVDIbyUuid(XenServerStorageProcessor.java:743)
Caused by: The uuid you supplied was invalid.
Unsupported data object (VOLUME,
org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@4ddd9112),
no need to delete from object in store ref table
com.cloud.exception.StorageUnavailableException: Resource
[StoragePool:6] is unreachable: Unable to create
Vol[5128|vm=5101|ROOT]:com.cloud.utils.exception.CloudRuntimeException:
Catch Exception com.xensource.xenapi.Types$UuidInvalid :VDI getByUuid
for uuid: 5e2f645f-9e5d-45df-a268-dbe9c6c9eef9 failed due to The uuid
you supplied was invalid.
It's also writing, that storage is unreachable Resource [StoragePool:6]
is unreachable: Unable to create
But as i wrote, we can create new diskdata and so on on this storage
without problem.
https://pastebin.com/HLt40fK0
Re: VR starting problem
Posted by he...@powershield.ru.
We re-checked in:
Global settings
Zone Settings
Cluster Settings
Again re-checked right now:
Global Settings: CPU, Memory and Storages set to 0.98
Zone Settings: Storage set to 0.98
Cluster Settings: CPU, Memory and Storages set to 0.98
We restart management server like 10 times.
We even restart all management server.
Anyway, the same problem:)
Paul Angus писал 2017-11-07 00:18:
> Yes, I see that, but as the message is still the same, you either
> changed a similar but different setting (there are a number of
> similarly named global settings) or you did not restart all of your
> management servers.
>
>
>
> Kind regards,
>
> Paul Angus
>
> paul.angus@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> @shapeblue
>
>
>
>
> -----Original Message-----
> From: hello@powershield.ru [mailto:hello@powershield.ru]
> Sent: 06 November 2017 21:15
> To: users@cloudstack.apache.org
> Subject: Re: VR starting problem
>
> As i wrote before, i already did it before first my e-mail here, but
> without luck:)
>
>
>
> Paul Angus писал 2017-11-07 00:12:
>> Ideally we'd need to start the log even earlier, but I can still see:
>> Insufficient space on pool: 6 since its usage percentage:
>> 0.8669012505498059 has crossed the
>> pool.storage.capacity.disablethreshold: 0.85
>>
>> If you increased pool.storage.capacity.disablethreshold to above 087,
>> then you need to restart your management server(s) for it to take
>> effect.
>>
>> It would be better though, if you add another primary storage pool, or
>> increase the storage in the existing pool. As increasing the threshold
>> isn't a long term solution.
>>
>>
>>
>>
>>
>>
>> Kind regards,
>>
>> Paul Angus
>>
>> paul.angus@shapeblue.com
>> www.shapeblue.com
>> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
>> @shapeblue
>>
>>
>>
>>
>> -----Original Message-----
>> From: hello@powershield.ru [mailto:hello@powershield.ru]
>> Sent: 06 November 2017 20:53
>> To: users@cloudstack.apache.org
>> Subject: Re: VR starting problem
>>
>> Hello,
>>
>> Here is some of log.
>>
>> https://pastebin.com/JU0qr69V
>>
>> Interested line: dataCenterId:: 2 // podId:: null // clusterId:: null
>>
>> As you can see, datacenter (zone) showing, but can't see pod and
>> cluster....
>>
>>
>> Paul Angus писал 2017-11-06 22:44:
>>> Please could you use something like pastebin (https://pastebin.com/)
>>> to paste larger sections of your logs.
>>> The section you have pasted is more likely from XenServer trying to
>>> remove a disk image that was never created in the first place. It
>>> may
>>> not be the root cause.
>>>
>>> Kind regards,
>>>
>>> Paul Angus
>>>
>>> paul.angus@shapeblue.com
>>> www.shapeblue.com
>>> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
>>> @shapeblue
>>>
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: hello@powershield.ru [mailto:hello@powershield.ru]
>>> Sent: 06 November 2017 18:14
>>> To: users@cloudstack.apache.org
>>> Subject: Re: VR starting problem
>>>
>>> We deleted system VM's and also, can't start.
>>>
>>> Now seeing the error:
>>>
>>>
>>> INFO [c.c.v.VirtualMachineManagerImpl]
>>> (Work-Job-Executor-11:ctx-eaf4ab20 job-97/job-14993 ctx-7391ff7c)
>>> (logid:66199d2c) Unable to contact resource.
>>> com.cloud.exception.StorageUnavailableException: Resource
>>> [StoragePool:6] is unreachable: Unable to create
>>> Vol[2034|vm=2010|ROOT]:com.cloud.utils.exception.CloudRuntimeException:
>>> Catch Exception com.xensource.xenapi.Types$UuidInvalid :VDI getByUuid
>>> for uuid: 5e2f645f-9e5d-45df-a268-dbe9c6c9eef9 failed due to The uuid
>>> you supplied was invalid.
>>>
>>> 2017-11-06 20:16:49,498 DEBUG [c.c.h.x.r.XenServerStorageProcessor]
>>> (DirectAgent-6:ctx-b7ab797e) (logid:d3084120) Catch Exception
>>> com.xensource.xenapi.Types$UuidInvalid :VDI getByUuid for uuid:
>>> 5e2f645f-9e5d-45df-a268-dbe9c6c9eef9 failed due to The uuid you
>>> supplied was invalid.
>>> 2017-11-06 20:16:49,501 WARN [c.c.h.x.r.XenServerStorageProcessor]
>>> (DirectAgent-6:ctx-b7ab797e) (logid:d3084120) Unable to create
>>> volume;
>>> Pool=volumeTO[uuid=efa30838-f8e4-4186-a299-d2ae716d9081|path=null|datastore=PrimaryDataStoreTO[uuid=SR-UK|name=null|id=6|pooltype=PreSetup]];
>>> Disk:
>>> com.cloud.utils.exception.CloudRuntimeException: Catch Exception
>>> com.xensource.xenapi.Types$UuidInvalid :VDI getByUuid for uuid:
>>> 5e2f645f-9e5d-45df-a268-dbe9c6c9eef9 failed due to The uuid you
>>> supplied was invalid.
>>>
>>>
>>>
>>>
>>>
>>> hello@powershield.ru писал 2017-11-06 20:48:
>>>> Changed few hours ago, and anyway still the same problem.
>>>>
>>>>
>>>>
>>>> Ivan Kudryavtsev писал 2017-11-06 20:40:
>>>>> Hi. Don't mix "used" and "allocated". Just change the specified
>>>>> threshold to 0.98 and go ahead.
>>>>>
>>>>> 7 нояб. 2017 г. 0:22 пользователь <he...@powershield.ru> написал:
>>>>>
>>>>>> No, i re-checked maybe 10 times. I just added some more space, but
>>>>>> the problem is same. All VM's working but if client restart - they
>>>>>> goes down, with the same error with insufficient storage.
>>>>>>
>>>>>>
>>>>>> Ivan Kudryavtsev писал 2017-11-06 20:16:
>>>>>>
>>>>>>> Could it be, that your storage, cluster, pod, zone is in the
>>>>>>> maintanence mode?
>>>>>>>
>>>>>>> 6 нояб. 2017 г. 23:51 пользователь "Ivan Kudryavtsev" <
>>>>>>> kudryavtsev_ia@bw-sw.com> написал:
>>>>>>>
>>>>>>> Hi. Try to find in early logs what exactly resource is
>>>>>>> insufficient.
>>>>>>> Try
>>>>>>>> to play with threshold of it.
>>>>>>>>
>>>>>>>> 6 нояб. 2017 г. 21:57 пользователь <he...@powershield.ru>
>>>>>>>> написал:
>>>>>>>>
>>>>>>>> Now, i see the problem, then VR restarting.
>>>>>>>>>
>>>>>>>>> 2017-11-06 11:54:15,821 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>>>> (API-Job-Executor-43:ctx-02214ed9 job-14455) (logid:18ad84ad)
>>>>>>>>> Complete async job-14455, jobStatus: FAILED, resultCode: 530,
>>>>>>>>> result:
>>>>>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>>>>>>>>> uuidList":[],"errorcode":530,"errortext":"Resource [Host:17] is
>>>>>>>>> unreachable: Host 17: Unable to start instance due to Unable to
>>>>>>>>> start
>>>>>>>>> VM:18985a14-3cb5-46d5-a3f0-3a065cd8b014 due to error in
>>>>>>>>> finalizeStart, not retrying"}
>>>>>>>>> 2017-11-06 15:47:27,814 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>>>> (API-Job-Executor-69:ctx-be0b5550 job-14663) (logid:8b609324)
>>>>>>>>> Complete async job-14663, jobStatus: FAILED, resultCode: 530,
>>>>>>>>> result:
>>>>>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>>>>>>>>> uuidList":[],"errorcode":530,"errortext":"Unable to start a VM
>>>>>>>>> due to insufficient capacity"}
>>>>>>>>> 2017-11-06 16:54:19,055 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>>>> (API-Job-Executor-70:ctx-deaaf8e8 job-14668) (logid:6a7700af)
>>>>>>>>> Complete async job-14668, jobStatus: FAILED, resultCode: 530,
>>>>>>>>> result:
>>>>>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>>>>>>>>> uuidList":[],"errorcode":530,"errortext":"Resource
>>>>>>>>> [DataCenter:2]
>>>>>>>>> is
>>>>>>>>> unreachable: Unable to reboot domR, it is not in right state
>>>>>>>>> Stopped"}
>>>>>>>>> 2017-11-06 16:58:54,235 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>>>> (API-Job-Executor-71:ctx-b26b3b33 job-14669) (logid:436e6183)
>>>>>>>>> Complete async job-14669, jobStatus: FAILED, resultCode: 530,
>>>>>>>>> result:
>>>>>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>>>>>>>>> uuidList":[],"errorcode":530,"errortext":"Unable to create a
>>>>>>>>> deployment for VM[DomainRouter|r-1919-VM]"}
>>>>>>>>>
>>>>>>>>> I'm pretty sure, what i have enough resources:
>>>>>>>>>
>>>>>>>>> Primary Storage Used
>>>>>>>>> Allocated: 4.22 TB / 4.88 TB
>>>>>>>>>
>>>>>>>>> Primary Storage Allocated
>>>>>>>>> Allocated: 3.02 TB / 4.88 TB
>>>>>>>>>
>>>>>>>>> Memory Allocated
>>>>>>>>> Allocated: 78.00 GB / 597.93 GB
>>>>>>>>>
>>>>>>>>> CPU Allocated
>>>>>>>>> Allocated: 81.00 GHz / 440.00 GHz
>>>>>>>>>
>>>>>>>>> Shared Network IPs
>>>>>>>>> Allocated: 34 / 253
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Ivan Kudryavtsev писал 2017-11-06 15:28:
>>>>>>>>>
>>>>>>>>> Hi, you should dive into management server logs to find
>>>>>>>>> relevant
>>>>>>>>>> messages.
>>>>>>>>>> Otherwise it's impossible to predict what's wrong with the
>>>>>>>>>> system.
>>>>>>>>>> Check
>>>>>>>>>> logs and find exceptions and other errornous stuff.
>>>>>>>>>>
>>>>>>>>>> 6 нояб. 2017 г. 18:14 пользователь <he...@powershield.ru>
>>>>>>>>>> написал:
>>>>>>>>>>
>>>>>>>>>> Hello,
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ACS: 4.8.0
>>>>>>>>>>> XenServer 6.5SP1
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> We have got some problem with additional space for our
>>>>>>>>>>> client.
>>>>>>>>>>>
>>>>>>>>>>> We are created new Disk Offering, then we tryed to attach for
>>>>>>>>>>> our client, but got error: Unable to find suitable primary
>>>>>>>>>>> storage when creating volume
>>>>>>>>>>>
>>>>>>>>>>> We checked datadisk and we see, there is not hypervisor
>>>>>>>>>>> (should
>>>>>>>>>>> it be showed? Now saying: None)
>>>>>>>>>>>
>>>>>>>>>>> We tryed to create new instance, but also unlucky.
>>>>>>>>>>>
>>>>>>>>>>> After it, we restarted VR and he can't start. He started and
>>>>>>>>>>> again goes to offline. We tryed to check from XenCenter and
>>>>>>>>>>> got
>>>>>>>>>>> this reason:
>>>>>>>>>>> apache2:
>>>>>>>>>>> Could not reliably determine the server's fully qualified
>>>>>>>>>>> domain name, using 127.0.0.1 for ServerName.
>>>>>>>>>>>
>>>>>>>>>>> We delete VR and now, we can not start it manually and he is
>>>>>>>>>>> not starting automatically about 40min.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
Re: VR starting problem
Posted by he...@powershield.ru.
Good morning for everyone,
we still have the problem, anyone have any ideas?
Primary storage is presetup'ed and can see it in GUI. In logs, we found
this:
StorageUnavailableException: Resource [StoragePool:6] is unreachable
We can't start systemvm's.
All logs:
https://pastebin.com/JU0qr69V
https://pastebin.com/A5Z4zA2U
Paul Angus писал 2017-11-07 00:18:
> Yes, I see that, but as the message is still the same, you either
> changed a similar but different setting (there are a number of
> similarly named global settings) or you did not restart all of your
> management servers.
>
>
>
> Kind regards,
>
> Paul Angus
>
> paul.angus@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> @shapeblue
>
>
>
>
> -----Original Message-----
> From: hello@powershield.ru [mailto:hello@powershield.ru]
> Sent: 06 November 2017 21:15
> To: users@cloudstack.apache.org
> Subject: Re: VR starting problem
>
> As i wrote before, i already did it before first my e-mail here, but
> without luck:)
>
>
>
> Paul Angus писал 2017-11-07 00:12:
>> Ideally we'd need to start the log even earlier, but I can still see:
>> Insufficient space on pool: 6 since its usage percentage:
>> 0.8669012505498059 has crossed the
>> pool.storage.capacity.disablethreshold: 0.85
>>
>> If you increased pool.storage.capacity.disablethreshold to above 087,
>> then you need to restart your management server(s) for it to take
>> effect.
>>
>> It would be better though, if you add another primary storage pool, or
>> increase the storage in the existing pool. As increasing the threshold
>> isn't a long term solution.
>>
>>
>>
>>
>>
>>
>> Kind regards,
>>
>> Paul Angus
>>
>> paul.angus@shapeblue.com
>> www.shapeblue.com
>> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
>> @shapeblue
>>
>>
>>
>>
>> -----Original Message-----
>> From: hello@powershield.ru [mailto:hello@powershield.ru]
>> Sent: 06 November 2017 20:53
>> To: users@cloudstack.apache.org
>> Subject: Re: VR starting problem
>>
>> Hello,
>>
>> Here is some of log.
>>
>> https://pastebin.com/JU0qr69V
>>
>> Interested line: dataCenterId:: 2 // podId:: null // clusterId:: null
>>
>> As you can see, datacenter (zone) showing, but can't see pod and
>> cluster....
>>
>>
>> Paul Angus писал 2017-11-06 22:44:
>>> Please could you use something like pastebin (https://pastebin.com/)
>>> to paste larger sections of your logs.
>>> The section you have pasted is more likely from XenServer trying to
>>> remove a disk image that was never created in the first place. It
>>> may
>>> not be the root cause.
>>>
>>> Kind regards,
>>>
>>> Paul Angus
>>>
>>> paul.angus@shapeblue.com
>>> www.shapeblue.com
>>> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
>>> @shapeblue
>>>
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: hello@powershield.ru [mailto:hello@powershield.ru]
>>> Sent: 06 November 2017 18:14
>>> To: users@cloudstack.apache.org
>>> Subject: Re: VR starting problem
>>>
>>> We deleted system VM's and also, can't start.
>>>
>>> Now seeing the error:
>>>
>>>
>>> INFO [c.c.v.VirtualMachineManagerImpl]
>>> (Work-Job-Executor-11:ctx-eaf4ab20 job-97/job-14993 ctx-7391ff7c)
>>> (logid:66199d2c) Unable to contact resource.
>>> com.cloud.exception.StorageUnavailableException: Resource
>>> [StoragePool:6] is unreachable: Unable to create
>>> Vol[2034|vm=2010|ROOT]:com.cloud.utils.exception.CloudRuntimeException:
>>> Catch Exception com.xensource.xenapi.Types$UuidInvalid :VDI getByUuid
>>> for uuid: 5e2f645f-9e5d-45df-a268-dbe9c6c9eef9 failed due to The uuid
>>> you supplied was invalid.
>>>
>>> 2017-11-06 20:16:49,498 DEBUG [c.c.h.x.r.XenServerStorageProcessor]
>>> (DirectAgent-6:ctx-b7ab797e) (logid:d3084120) Catch Exception
>>> com.xensource.xenapi.Types$UuidInvalid :VDI getByUuid for uuid:
>>> 5e2f645f-9e5d-45df-a268-dbe9c6c9eef9 failed due to The uuid you
>>> supplied was invalid.
>>> 2017-11-06 20:16:49,501 WARN [c.c.h.x.r.XenServerStorageProcessor]
>>> (DirectAgent-6:ctx-b7ab797e) (logid:d3084120) Unable to create
>>> volume;
>>> Pool=volumeTO[uuid=efa30838-f8e4-4186-a299-d2ae716d9081|path=null|datastore=PrimaryDataStoreTO[uuid=SR-UK|name=null|id=6|pooltype=PreSetup]];
>>> Disk:
>>> com.cloud.utils.exception.CloudRuntimeException: Catch Exception
>>> com.xensource.xenapi.Types$UuidInvalid :VDI getByUuid for uuid:
>>> 5e2f645f-9e5d-45df-a268-dbe9c6c9eef9 failed due to The uuid you
>>> supplied was invalid.
>>>
>>>
>>>
>>>
>>>
>>> hello@powershield.ru писал 2017-11-06 20:48:
>>>> Changed few hours ago, and anyway still the same problem.
>>>>
>>>>
>>>>
>>>> Ivan Kudryavtsev писал 2017-11-06 20:40:
>>>>> Hi. Don't mix "used" and "allocated". Just change the specified
>>>>> threshold to 0.98 and go ahead.
>>>>>
>>>>> 7 нояб. 2017 г. 0:22 пользователь <he...@powershield.ru> написал:
>>>>>
>>>>>> No, i re-checked maybe 10 times. I just added some more space, but
>>>>>> the problem is same. All VM's working but if client restart - they
>>>>>> goes down, with the same error with insufficient storage.
>>>>>>
>>>>>>
>>>>>> Ivan Kudryavtsev писал 2017-11-06 20:16:
>>>>>>
>>>>>>> Could it be, that your storage, cluster, pod, zone is in the
>>>>>>> maintanence mode?
>>>>>>>
>>>>>>> 6 нояб. 2017 г. 23:51 пользователь "Ivan Kudryavtsev" <
>>>>>>> kudryavtsev_ia@bw-sw.com> написал:
>>>>>>>
>>>>>>> Hi. Try to find in early logs what exactly resource is
>>>>>>> insufficient.
>>>>>>> Try
>>>>>>>> to play with threshold of it.
>>>>>>>>
>>>>>>>> 6 нояб. 2017 г. 21:57 пользователь <he...@powershield.ru>
>>>>>>>> написал:
>>>>>>>>
>>>>>>>> Now, i see the problem, then VR restarting.
>>>>>>>>>
>>>>>>>>> 2017-11-06 11:54:15,821 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>>>> (API-Job-Executor-43:ctx-02214ed9 job-14455) (logid:18ad84ad)
>>>>>>>>> Complete async job-14455, jobStatus: FAILED, resultCode: 530,
>>>>>>>>> result:
>>>>>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>>>>>>>>> uuidList":[],"errorcode":530,"errortext":"Resource [Host:17] is
>>>>>>>>> unreachable: Host 17: Unable to start instance due to Unable to
>>>>>>>>> start
>>>>>>>>> VM:18985a14-3cb5-46d5-a3f0-3a065cd8b014 due to error in
>>>>>>>>> finalizeStart, not retrying"}
>>>>>>>>> 2017-11-06 15:47:27,814 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>>>> (API-Job-Executor-69:ctx-be0b5550 job-14663) (logid:8b609324)
>>>>>>>>> Complete async job-14663, jobStatus: FAILED, resultCode: 530,
>>>>>>>>> result:
>>>>>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>>>>>>>>> uuidList":[],"errorcode":530,"errortext":"Unable to start a VM
>>>>>>>>> due to insufficient capacity"}
>>>>>>>>> 2017-11-06 16:54:19,055 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>>>> (API-Job-Executor-70:ctx-deaaf8e8 job-14668) (logid:6a7700af)
>>>>>>>>> Complete async job-14668, jobStatus: FAILED, resultCode: 530,
>>>>>>>>> result:
>>>>>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>>>>>>>>> uuidList":[],"errorcode":530,"errortext":"Resource
>>>>>>>>> [DataCenter:2]
>>>>>>>>> is
>>>>>>>>> unreachable: Unable to reboot domR, it is not in right state
>>>>>>>>> Stopped"}
>>>>>>>>> 2017-11-06 16:58:54,235 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>>>> (API-Job-Executor-71:ctx-b26b3b33 job-14669) (logid:436e6183)
>>>>>>>>> Complete async job-14669, jobStatus: FAILED, resultCode: 530,
>>>>>>>>> result:
>>>>>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>>>>>>>>> uuidList":[],"errorcode":530,"errortext":"Unable to create a
>>>>>>>>> deployment for VM[DomainRouter|r-1919-VM]"}
>>>>>>>>>
>>>>>>>>> I'm pretty sure, what i have enough resources:
>>>>>>>>>
>>>>>>>>> Primary Storage Used
>>>>>>>>> Allocated: 4.22 TB / 4.88 TB
>>>>>>>>>
>>>>>>>>> Primary Storage Allocated
>>>>>>>>> Allocated: 3.02 TB / 4.88 TB
>>>>>>>>>
>>>>>>>>> Memory Allocated
>>>>>>>>> Allocated: 78.00 GB / 597.93 GB
>>>>>>>>>
>>>>>>>>> CPU Allocated
>>>>>>>>> Allocated: 81.00 GHz / 440.00 GHz
>>>>>>>>>
>>>>>>>>> Shared Network IPs
>>>>>>>>> Allocated: 34 / 253
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Ivan Kudryavtsev писал 2017-11-06 15:28:
>>>>>>>>>
>>>>>>>>> Hi, you should dive into management server logs to find
>>>>>>>>> relevant
>>>>>>>>>> messages.
>>>>>>>>>> Otherwise it's impossible to predict what's wrong with the
>>>>>>>>>> system.
>>>>>>>>>> Check
>>>>>>>>>> logs and find exceptions and other errornous stuff.
>>>>>>>>>>
>>>>>>>>>> 6 нояб. 2017 г. 18:14 пользователь <he...@powershield.ru>
>>>>>>>>>> написал:
>>>>>>>>>>
>>>>>>>>>> Hello,
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ACS: 4.8.0
>>>>>>>>>>> XenServer 6.5SP1
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> We have got some problem with additional space for our
>>>>>>>>>>> client.
>>>>>>>>>>>
>>>>>>>>>>> We are created new Disk Offering, then we tryed to attach for
>>>>>>>>>>> our client, but got error: Unable to find suitable primary
>>>>>>>>>>> storage when creating volume
>>>>>>>>>>>
>>>>>>>>>>> We checked datadisk and we see, there is not hypervisor
>>>>>>>>>>> (should
>>>>>>>>>>> it be showed? Now saying: None)
>>>>>>>>>>>
>>>>>>>>>>> We tryed to create new instance, but also unlucky.
>>>>>>>>>>>
>>>>>>>>>>> After it, we restarted VR and he can't start. He started and
>>>>>>>>>>> again goes to offline. We tryed to check from XenCenter and
>>>>>>>>>>> got
>>>>>>>>>>> this reason:
>>>>>>>>>>> apache2:
>>>>>>>>>>> Could not reliably determine the server's fully qualified
>>>>>>>>>>> domain name, using 127.0.0.1 for ServerName.
>>>>>>>>>>>
>>>>>>>>>>> We delete VR and now, we can not start it manually and he is
>>>>>>>>>>> not starting automatically about 40min.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
RE: VR starting problem
Posted by Paul Angus <pa...@shapeblue.com>.
Yes, I see that, but as the message is still the same, you either changed a similar but different setting (there are a number of similarly named global settings) or you did not restart all of your management servers.
Kind regards,
Paul Angus
paul.angus@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
-----Original Message-----
From: hello@powershield.ru [mailto:hello@powershield.ru]
Sent: 06 November 2017 21:15
To: users@cloudstack.apache.org
Subject: Re: VR starting problem
As i wrote before, i already did it before first my e-mail here, but without luck:)
Paul Angus писал 2017-11-07 00:12:
> Ideally we'd need to start the log even earlier, but I can still see:
> Insufficient space on pool: 6 since its usage percentage:
> 0.8669012505498059 has crossed the
> pool.storage.capacity.disablethreshold: 0.85
>
> If you increased pool.storage.capacity.disablethreshold to above 087,
> then you need to restart your management server(s) for it to take
> effect.
>
> It would be better though, if you add another primary storage pool, or
> increase the storage in the existing pool. As increasing the threshold
> isn't a long term solution.
>
>
>
>
>
>
> Kind regards,
>
> Paul Angus
>
> paul.angus@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> @shapeblue
>
>
>
>
> -----Original Message-----
> From: hello@powershield.ru [mailto:hello@powershield.ru]
> Sent: 06 November 2017 20:53
> To: users@cloudstack.apache.org
> Subject: Re: VR starting problem
>
> Hello,
>
> Here is some of log.
>
> https://pastebin.com/JU0qr69V
>
> Interested line: dataCenterId:: 2 // podId:: null // clusterId:: null
>
> As you can see, datacenter (zone) showing, but can't see pod and
> cluster....
>
>
> Paul Angus писал 2017-11-06 22:44:
>> Please could you use something like pastebin (https://pastebin.com/)
>> to paste larger sections of your logs.
>> The section you have pasted is more likely from XenServer trying to
>> remove a disk image that was never created in the first place. It may
>> not be the root cause.
>>
>> Kind regards,
>>
>> Paul Angus
>>
>> paul.angus@shapeblue.com
>> www.shapeblue.com
>> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
>> @shapeblue
>>
>>
>>
>>
>> -----Original Message-----
>> From: hello@powershield.ru [mailto:hello@powershield.ru]
>> Sent: 06 November 2017 18:14
>> To: users@cloudstack.apache.org
>> Subject: Re: VR starting problem
>>
>> We deleted system VM's and also, can't start.
>>
>> Now seeing the error:
>>
>>
>> INFO [c.c.v.VirtualMachineManagerImpl]
>> (Work-Job-Executor-11:ctx-eaf4ab20 job-97/job-14993 ctx-7391ff7c)
>> (logid:66199d2c) Unable to contact resource.
>> com.cloud.exception.StorageUnavailableException: Resource
>> [StoragePool:6] is unreachable: Unable to create
>> Vol[2034|vm=2010|ROOT]:com.cloud.utils.exception.CloudRuntimeException:
>> Catch Exception com.xensource.xenapi.Types$UuidInvalid :VDI getByUuid
>> for uuid: 5e2f645f-9e5d-45df-a268-dbe9c6c9eef9 failed due to The uuid
>> you supplied was invalid.
>>
>> 2017-11-06 20:16:49,498 DEBUG [c.c.h.x.r.XenServerStorageProcessor]
>> (DirectAgent-6:ctx-b7ab797e) (logid:d3084120) Catch Exception
>> com.xensource.xenapi.Types$UuidInvalid :VDI getByUuid for uuid:
>> 5e2f645f-9e5d-45df-a268-dbe9c6c9eef9 failed due to The uuid you
>> supplied was invalid.
>> 2017-11-06 20:16:49,501 WARN [c.c.h.x.r.XenServerStorageProcessor]
>> (DirectAgent-6:ctx-b7ab797e) (logid:d3084120) Unable to create volume;
>> Pool=volumeTO[uuid=efa30838-f8e4-4186-a299-d2ae716d9081|path=null|datastore=PrimaryDataStoreTO[uuid=SR-UK|name=null|id=6|pooltype=PreSetup]];
>> Disk:
>> com.cloud.utils.exception.CloudRuntimeException: Catch Exception
>> com.xensource.xenapi.Types$UuidInvalid :VDI getByUuid for uuid:
>> 5e2f645f-9e5d-45df-a268-dbe9c6c9eef9 failed due to The uuid you
>> supplied was invalid.
>>
>>
>>
>>
>>
>> hello@powershield.ru писал 2017-11-06 20:48:
>>> Changed few hours ago, and anyway still the same problem.
>>>
>>>
>>>
>>> Ivan Kudryavtsev писал 2017-11-06 20:40:
>>>> Hi. Don't mix "used" and "allocated". Just change the specified
>>>> threshold to 0.98 and go ahead.
>>>>
>>>> 7 нояб. 2017 г. 0:22 пользователь <he...@powershield.ru> написал:
>>>>
>>>>> No, i re-checked maybe 10 times. I just added some more space, but
>>>>> the problem is same. All VM's working but if client restart - they
>>>>> goes down, with the same error with insufficient storage.
>>>>>
>>>>>
>>>>> Ivan Kudryavtsev писал 2017-11-06 20:16:
>>>>>
>>>>>> Could it be, that your storage, cluster, pod, zone is in the
>>>>>> maintanence mode?
>>>>>>
>>>>>> 6 нояб. 2017 г. 23:51 пользователь "Ivan Kudryavtsev" <
>>>>>> kudryavtsev_ia@bw-sw.com> написал:
>>>>>>
>>>>>> Hi. Try to find in early logs what exactly resource is
>>>>>> insufficient.
>>>>>> Try
>>>>>>> to play with threshold of it.
>>>>>>>
>>>>>>> 6 нояб. 2017 г. 21:57 пользователь <he...@powershield.ru>
>>>>>>> написал:
>>>>>>>
>>>>>>> Now, i see the problem, then VR restarting.
>>>>>>>>
>>>>>>>> 2017-11-06 11:54:15,821 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>>> (API-Job-Executor-43:ctx-02214ed9 job-14455) (logid:18ad84ad)
>>>>>>>> Complete async job-14455, jobStatus: FAILED, resultCode: 530,
>>>>>>>> result:
>>>>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>>>>>>>> uuidList":[],"errorcode":530,"errortext":"Resource [Host:17] is
>>>>>>>> unreachable: Host 17: Unable to start instance due to Unable to
>>>>>>>> start
>>>>>>>> VM:18985a14-3cb5-46d5-a3f0-3a065cd8b014 due to error in
>>>>>>>> finalizeStart, not retrying"}
>>>>>>>> 2017-11-06 15:47:27,814 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>>> (API-Job-Executor-69:ctx-be0b5550 job-14663) (logid:8b609324)
>>>>>>>> Complete async job-14663, jobStatus: FAILED, resultCode: 530,
>>>>>>>> result:
>>>>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>>>>>>>> uuidList":[],"errorcode":530,"errortext":"Unable to start a VM
>>>>>>>> due to insufficient capacity"}
>>>>>>>> 2017-11-06 16:54:19,055 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>>> (API-Job-Executor-70:ctx-deaaf8e8 job-14668) (logid:6a7700af)
>>>>>>>> Complete async job-14668, jobStatus: FAILED, resultCode: 530,
>>>>>>>> result:
>>>>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>>>>>>>> uuidList":[],"errorcode":530,"errortext":"Resource
>>>>>>>> [DataCenter:2]
>>>>>>>> is
>>>>>>>> unreachable: Unable to reboot domR, it is not in right state
>>>>>>>> Stopped"}
>>>>>>>> 2017-11-06 16:58:54,235 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>>> (API-Job-Executor-71:ctx-b26b3b33 job-14669) (logid:436e6183)
>>>>>>>> Complete async job-14669, jobStatus: FAILED, resultCode: 530,
>>>>>>>> result:
>>>>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>>>>>>>> uuidList":[],"errorcode":530,"errortext":"Unable to create a
>>>>>>>> deployment for VM[DomainRouter|r-1919-VM]"}
>>>>>>>>
>>>>>>>> I'm pretty sure, what i have enough resources:
>>>>>>>>
>>>>>>>> Primary Storage Used
>>>>>>>> Allocated: 4.22 TB / 4.88 TB
>>>>>>>>
>>>>>>>> Primary Storage Allocated
>>>>>>>> Allocated: 3.02 TB / 4.88 TB
>>>>>>>>
>>>>>>>> Memory Allocated
>>>>>>>> Allocated: 78.00 GB / 597.93 GB
>>>>>>>>
>>>>>>>> CPU Allocated
>>>>>>>> Allocated: 81.00 GHz / 440.00 GHz
>>>>>>>>
>>>>>>>> Shared Network IPs
>>>>>>>> Allocated: 34 / 253
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Ivan Kudryavtsev писал 2017-11-06 15:28:
>>>>>>>>
>>>>>>>> Hi, you should dive into management server logs to find relevant
>>>>>>>>> messages.
>>>>>>>>> Otherwise it's impossible to predict what's wrong with the
>>>>>>>>> system.
>>>>>>>>> Check
>>>>>>>>> logs and find exceptions and other errornous stuff.
>>>>>>>>>
>>>>>>>>> 6 нояб. 2017 г. 18:14 пользователь <he...@powershield.ru>
>>>>>>>>> написал:
>>>>>>>>>
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ACS: 4.8.0
>>>>>>>>>> XenServer 6.5SP1
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> We have got some problem with additional space for our client.
>>>>>>>>>>
>>>>>>>>>> We are created new Disk Offering, then we tryed to attach for
>>>>>>>>>> our client, but got error: Unable to find suitable primary
>>>>>>>>>> storage when creating volume
>>>>>>>>>>
>>>>>>>>>> We checked datadisk and we see, there is not hypervisor
>>>>>>>>>> (should
>>>>>>>>>> it be showed? Now saying: None)
>>>>>>>>>>
>>>>>>>>>> We tryed to create new instance, but also unlucky.
>>>>>>>>>>
>>>>>>>>>> After it, we restarted VR and he can't start. He started and
>>>>>>>>>> again goes to offline. We tryed to check from XenCenter and
>>>>>>>>>> got
>>>>>>>>>> this reason:
>>>>>>>>>> apache2:
>>>>>>>>>> Could not reliably determine the server's fully qualified
>>>>>>>>>> domain name, using 127.0.0.1 for ServerName.
>>>>>>>>>>
>>>>>>>>>> We delete VR and now, we can not start it manually and he is
>>>>>>>>>> not starting automatically about 40min.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
Re: VR starting problem
Posted by he...@powershield.ru.
As i wrote before, i already did it before first my e-mail here, but
without luck:)
Paul Angus писал 2017-11-07 00:12:
> Ideally we'd need to start the log even earlier, but I can still see:
> Insufficient space on pool: 6 since its usage percentage:
> 0.8669012505498059 has crossed the
> pool.storage.capacity.disablethreshold: 0.85
>
> If you increased pool.storage.capacity.disablethreshold to above 087,
> then you need to restart your management server(s) for it to take
> effect.
>
> It would be better though, if you add another primary storage pool, or
> increase the storage in the existing pool. As increasing the threshold
> isn't a long term solution.
>
>
>
>
>
>
> Kind regards,
>
> Paul Angus
>
> paul.angus@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> @shapeblue
>
>
>
>
> -----Original Message-----
> From: hello@powershield.ru [mailto:hello@powershield.ru]
> Sent: 06 November 2017 20:53
> To: users@cloudstack.apache.org
> Subject: Re: VR starting problem
>
> Hello,
>
> Here is some of log.
>
> https://pastebin.com/JU0qr69V
>
> Interested line: dataCenterId:: 2 // podId:: null // clusterId:: null
>
> As you can see, datacenter (zone) showing, but can't see pod and
> cluster....
>
>
> Paul Angus писал 2017-11-06 22:44:
>> Please could you use something like pastebin (https://pastebin.com/)
>> to paste larger sections of your logs.
>> The section you have pasted is more likely from XenServer trying to
>> remove a disk image that was never created in the first place. It may
>> not be the root cause.
>>
>> Kind regards,
>>
>> Paul Angus
>>
>> paul.angus@shapeblue.com
>> www.shapeblue.com
>> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
>> @shapeblue
>>
>>
>>
>>
>> -----Original Message-----
>> From: hello@powershield.ru [mailto:hello@powershield.ru]
>> Sent: 06 November 2017 18:14
>> To: users@cloudstack.apache.org
>> Subject: Re: VR starting problem
>>
>> We deleted system VM's and also, can't start.
>>
>> Now seeing the error:
>>
>>
>> INFO [c.c.v.VirtualMachineManagerImpl]
>> (Work-Job-Executor-11:ctx-eaf4ab20 job-97/job-14993 ctx-7391ff7c)
>> (logid:66199d2c) Unable to contact resource.
>> com.cloud.exception.StorageUnavailableException: Resource
>> [StoragePool:6] is unreachable: Unable to create
>> Vol[2034|vm=2010|ROOT]:com.cloud.utils.exception.CloudRuntimeException:
>> Catch Exception com.xensource.xenapi.Types$UuidInvalid :VDI getByUuid
>> for uuid: 5e2f645f-9e5d-45df-a268-dbe9c6c9eef9 failed due to The uuid
>> you supplied was invalid.
>>
>> 2017-11-06 20:16:49,498 DEBUG [c.c.h.x.r.XenServerStorageProcessor]
>> (DirectAgent-6:ctx-b7ab797e) (logid:d3084120) Catch Exception
>> com.xensource.xenapi.Types$UuidInvalid :VDI getByUuid for uuid:
>> 5e2f645f-9e5d-45df-a268-dbe9c6c9eef9 failed due to The uuid you
>> supplied was invalid.
>> 2017-11-06 20:16:49,501 WARN [c.c.h.x.r.XenServerStorageProcessor]
>> (DirectAgent-6:ctx-b7ab797e) (logid:d3084120) Unable to create volume;
>> Pool=volumeTO[uuid=efa30838-f8e4-4186-a299-d2ae716d9081|path=null|datastore=PrimaryDataStoreTO[uuid=SR-UK|name=null|id=6|pooltype=PreSetup]];
>> Disk:
>> com.cloud.utils.exception.CloudRuntimeException: Catch Exception
>> com.xensource.xenapi.Types$UuidInvalid :VDI getByUuid for uuid:
>> 5e2f645f-9e5d-45df-a268-dbe9c6c9eef9 failed due to The uuid you
>> supplied was invalid.
>>
>>
>>
>>
>>
>> hello@powershield.ru писал 2017-11-06 20:48:
>>> Changed few hours ago, and anyway still the same problem.
>>>
>>>
>>>
>>> Ivan Kudryavtsev писал 2017-11-06 20:40:
>>>> Hi. Don't mix "used" and "allocated". Just change the specified
>>>> threshold to 0.98 and go ahead.
>>>>
>>>> 7 нояб. 2017 г. 0:22 пользователь <he...@powershield.ru> написал:
>>>>
>>>>> No, i re-checked maybe 10 times. I just added some more space, but
>>>>> the problem is same. All VM's working but if client restart - they
>>>>> goes down, with the same error with insufficient storage.
>>>>>
>>>>>
>>>>> Ivan Kudryavtsev писал 2017-11-06 20:16:
>>>>>
>>>>>> Could it be, that your storage, cluster, pod, zone is in the
>>>>>> maintanence mode?
>>>>>>
>>>>>> 6 нояб. 2017 г. 23:51 пользователь "Ivan Kudryavtsev" <
>>>>>> kudryavtsev_ia@bw-sw.com> написал:
>>>>>>
>>>>>> Hi. Try to find in early logs what exactly resource is
>>>>>> insufficient.
>>>>>> Try
>>>>>>> to play with threshold of it.
>>>>>>>
>>>>>>> 6 нояб. 2017 г. 21:57 пользователь <he...@powershield.ru>
>>>>>>> написал:
>>>>>>>
>>>>>>> Now, i see the problem, then VR restarting.
>>>>>>>>
>>>>>>>> 2017-11-06 11:54:15,821 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>>> (API-Job-Executor-43:ctx-02214ed9 job-14455) (logid:18ad84ad)
>>>>>>>> Complete async job-14455, jobStatus: FAILED, resultCode: 530,
>>>>>>>> result:
>>>>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>>>>>>>> uuidList":[],"errorcode":530,"errortext":"Resource [Host:17] is
>>>>>>>> unreachable: Host 17: Unable to start instance due to Unable to
>>>>>>>> start
>>>>>>>> VM:18985a14-3cb5-46d5-a3f0-3a065cd8b014 due to error in
>>>>>>>> finalizeStart, not retrying"}
>>>>>>>> 2017-11-06 15:47:27,814 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>>> (API-Job-Executor-69:ctx-be0b5550 job-14663) (logid:8b609324)
>>>>>>>> Complete async job-14663, jobStatus: FAILED, resultCode: 530,
>>>>>>>> result:
>>>>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>>>>>>>> uuidList":[],"errorcode":530,"errortext":"Unable to start a VM
>>>>>>>> due to insufficient capacity"}
>>>>>>>> 2017-11-06 16:54:19,055 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>>> (API-Job-Executor-70:ctx-deaaf8e8 job-14668) (logid:6a7700af)
>>>>>>>> Complete async job-14668, jobStatus: FAILED, resultCode: 530,
>>>>>>>> result:
>>>>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>>>>>>>> uuidList":[],"errorcode":530,"errortext":"Resource
>>>>>>>> [DataCenter:2]
>>>>>>>> is
>>>>>>>> unreachable: Unable to reboot domR, it is not in right state
>>>>>>>> Stopped"}
>>>>>>>> 2017-11-06 16:58:54,235 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>>> (API-Job-Executor-71:ctx-b26b3b33 job-14669) (logid:436e6183)
>>>>>>>> Complete async job-14669, jobStatus: FAILED, resultCode: 530,
>>>>>>>> result:
>>>>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>>>>>>>> uuidList":[],"errorcode":530,"errortext":"Unable to create a
>>>>>>>> deployment for VM[DomainRouter|r-1919-VM]"}
>>>>>>>>
>>>>>>>> I'm pretty sure, what i have enough resources:
>>>>>>>>
>>>>>>>> Primary Storage Used
>>>>>>>> Allocated: 4.22 TB / 4.88 TB
>>>>>>>>
>>>>>>>> Primary Storage Allocated
>>>>>>>> Allocated: 3.02 TB / 4.88 TB
>>>>>>>>
>>>>>>>> Memory Allocated
>>>>>>>> Allocated: 78.00 GB / 597.93 GB
>>>>>>>>
>>>>>>>> CPU Allocated
>>>>>>>> Allocated: 81.00 GHz / 440.00 GHz
>>>>>>>>
>>>>>>>> Shared Network IPs
>>>>>>>> Allocated: 34 / 253
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Ivan Kudryavtsev писал 2017-11-06 15:28:
>>>>>>>>
>>>>>>>> Hi, you should dive into management server logs to find relevant
>>>>>>>>> messages.
>>>>>>>>> Otherwise it's impossible to predict what's wrong with the
>>>>>>>>> system.
>>>>>>>>> Check
>>>>>>>>> logs and find exceptions and other errornous stuff.
>>>>>>>>>
>>>>>>>>> 6 нояб. 2017 г. 18:14 пользователь <he...@powershield.ru>
>>>>>>>>> написал:
>>>>>>>>>
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ACS: 4.8.0
>>>>>>>>>> XenServer 6.5SP1
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> We have got some problem with additional space for our client.
>>>>>>>>>>
>>>>>>>>>> We are created new Disk Offering, then we tryed to attach for
>>>>>>>>>> our client, but got error: Unable to find suitable primary
>>>>>>>>>> storage when creating volume
>>>>>>>>>>
>>>>>>>>>> We checked datadisk and we see, there is not hypervisor
>>>>>>>>>> (should
>>>>>>>>>> it be showed? Now saying: None)
>>>>>>>>>>
>>>>>>>>>> We tryed to create new instance, but also unlucky.
>>>>>>>>>>
>>>>>>>>>> After it, we restarted VR and he can't start. He started and
>>>>>>>>>> again goes to offline. We tryed to check from XenCenter and
>>>>>>>>>> got
>>>>>>>>>> this reason:
>>>>>>>>>> apache2:
>>>>>>>>>> Could not reliably determine the server's fully qualified
>>>>>>>>>> domain name, using 127.0.0.1 for ServerName.
>>>>>>>>>>
>>>>>>>>>> We delete VR and now, we can not start it manually and he is
>>>>>>>>>> not starting automatically about 40min.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
RE: VR starting problem
Posted by Paul Angus <pa...@shapeblue.com>.
Ideally we'd need to start the log even earlier, but I can still see:
Insufficient space on pool: 6 since its usage percentage: 0.8669012505498059 has crossed the pool.storage.capacity.disablethreshold: 0.85
If you increased pool.storage.capacity.disablethreshold to above 087, then you need to restart your management server(s) for it to take effect.
It would be better though, if you add another primary storage pool, or increase the storage in the existing pool. As increasing the threshold isn't a long term solution.
Kind regards,
Paul Angus
paul.angus@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
-----Original Message-----
From: hello@powershield.ru [mailto:hello@powershield.ru]
Sent: 06 November 2017 20:53
To: users@cloudstack.apache.org
Subject: Re: VR starting problem
Hello,
Here is some of log.
https://pastebin.com/JU0qr69V
Interested line: dataCenterId:: 2 // podId:: null // clusterId:: null
As you can see, datacenter (zone) showing, but can't see pod and cluster....
Paul Angus писал 2017-11-06 22:44:
> Please could you use something like pastebin (https://pastebin.com/)
> to paste larger sections of your logs.
> The section you have pasted is more likely from XenServer trying to
> remove a disk image that was never created in the first place. It may
> not be the root cause.
>
> Kind regards,
>
> Paul Angus
>
> paul.angus@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> @shapeblue
>
>
>
>
> -----Original Message-----
> From: hello@powershield.ru [mailto:hello@powershield.ru]
> Sent: 06 November 2017 18:14
> To: users@cloudstack.apache.org
> Subject: Re: VR starting problem
>
> We deleted system VM's and also, can't start.
>
> Now seeing the error:
>
>
> INFO [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-11:ctx-eaf4ab20 job-97/job-14993 ctx-7391ff7c)
> (logid:66199d2c) Unable to contact resource.
> com.cloud.exception.StorageUnavailableException: Resource
> [StoragePool:6] is unreachable: Unable to create
> Vol[2034|vm=2010|ROOT]:com.cloud.utils.exception.CloudRuntimeException:
> Catch Exception com.xensource.xenapi.Types$UuidInvalid :VDI getByUuid
> for uuid: 5e2f645f-9e5d-45df-a268-dbe9c6c9eef9 failed due to The uuid
> you supplied was invalid.
>
> 2017-11-06 20:16:49,498 DEBUG [c.c.h.x.r.XenServerStorageProcessor]
> (DirectAgent-6:ctx-b7ab797e) (logid:d3084120) Catch Exception
> com.xensource.xenapi.Types$UuidInvalid :VDI getByUuid for uuid:
> 5e2f645f-9e5d-45df-a268-dbe9c6c9eef9 failed due to The uuid you
> supplied was invalid.
> 2017-11-06 20:16:49,501 WARN [c.c.h.x.r.XenServerStorageProcessor]
> (DirectAgent-6:ctx-b7ab797e) (logid:d3084120) Unable to create volume;
> Pool=volumeTO[uuid=efa30838-f8e4-4186-a299-d2ae716d9081|path=null|datastore=PrimaryDataStoreTO[uuid=SR-UK|name=null|id=6|pooltype=PreSetup]];
> Disk:
> com.cloud.utils.exception.CloudRuntimeException: Catch Exception
> com.xensource.xenapi.Types$UuidInvalid :VDI getByUuid for uuid:
> 5e2f645f-9e5d-45df-a268-dbe9c6c9eef9 failed due to The uuid you
> supplied was invalid.
>
>
>
>
>
> hello@powershield.ru писал 2017-11-06 20:48:
>> Changed few hours ago, and anyway still the same problem.
>>
>>
>>
>> Ivan Kudryavtsev писал 2017-11-06 20:40:
>>> Hi. Don't mix "used" and "allocated". Just change the specified
>>> threshold to 0.98 and go ahead.
>>>
>>> 7 нояб. 2017 г. 0:22 пользователь <he...@powershield.ru> написал:
>>>
>>>> No, i re-checked maybe 10 times. I just added some more space, but
>>>> the problem is same. All VM's working but if client restart - they
>>>> goes down, with the same error with insufficient storage.
>>>>
>>>>
>>>> Ivan Kudryavtsev писал 2017-11-06 20:16:
>>>>
>>>>> Could it be, that your storage, cluster, pod, zone is in the
>>>>> maintanence mode?
>>>>>
>>>>> 6 нояб. 2017 г. 23:51 пользователь "Ivan Kudryavtsev" <
>>>>> kudryavtsev_ia@bw-sw.com> написал:
>>>>>
>>>>> Hi. Try to find in early logs what exactly resource is
>>>>> insufficient.
>>>>> Try
>>>>>> to play with threshold of it.
>>>>>>
>>>>>> 6 нояб. 2017 г. 21:57 пользователь <he...@powershield.ru> написал:
>>>>>>
>>>>>> Now, i see the problem, then VR restarting.
>>>>>>>
>>>>>>> 2017-11-06 11:54:15,821 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>> (API-Job-Executor-43:ctx-02214ed9 job-14455) (logid:18ad84ad)
>>>>>>> Complete async job-14455, jobStatus: FAILED, resultCode: 530,
>>>>>>> result:
>>>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>>>>>>> uuidList":[],"errorcode":530,"errortext":"Resource [Host:17] is
>>>>>>> unreachable: Host 17: Unable to start instance due to Unable to
>>>>>>> start
>>>>>>> VM:18985a14-3cb5-46d5-a3f0-3a065cd8b014 due to error in
>>>>>>> finalizeStart, not retrying"}
>>>>>>> 2017-11-06 15:47:27,814 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>> (API-Job-Executor-69:ctx-be0b5550 job-14663) (logid:8b609324)
>>>>>>> Complete async job-14663, jobStatus: FAILED, resultCode: 530,
>>>>>>> result:
>>>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>>>>>>> uuidList":[],"errorcode":530,"errortext":"Unable to start a VM
>>>>>>> due to insufficient capacity"}
>>>>>>> 2017-11-06 16:54:19,055 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>> (API-Job-Executor-70:ctx-deaaf8e8 job-14668) (logid:6a7700af)
>>>>>>> Complete async job-14668, jobStatus: FAILED, resultCode: 530,
>>>>>>> result:
>>>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>>>>>>> uuidList":[],"errorcode":530,"errortext":"Resource [DataCenter:2]
>>>>>>> is
>>>>>>> unreachable: Unable to reboot domR, it is not in right state
>>>>>>> Stopped"}
>>>>>>> 2017-11-06 16:58:54,235 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>> (API-Job-Executor-71:ctx-b26b3b33 job-14669) (logid:436e6183)
>>>>>>> Complete async job-14669, jobStatus: FAILED, resultCode: 530,
>>>>>>> result:
>>>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>>>>>>> uuidList":[],"errorcode":530,"errortext":"Unable to create a
>>>>>>> deployment for VM[DomainRouter|r-1919-VM]"}
>>>>>>>
>>>>>>> I'm pretty sure, what i have enough resources:
>>>>>>>
>>>>>>> Primary Storage Used
>>>>>>> Allocated: 4.22 TB / 4.88 TB
>>>>>>>
>>>>>>> Primary Storage Allocated
>>>>>>> Allocated: 3.02 TB / 4.88 TB
>>>>>>>
>>>>>>> Memory Allocated
>>>>>>> Allocated: 78.00 GB / 597.93 GB
>>>>>>>
>>>>>>> CPU Allocated
>>>>>>> Allocated: 81.00 GHz / 440.00 GHz
>>>>>>>
>>>>>>> Shared Network IPs
>>>>>>> Allocated: 34 / 253
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Ivan Kudryavtsev писал 2017-11-06 15:28:
>>>>>>>
>>>>>>> Hi, you should dive into management server logs to find relevant
>>>>>>>> messages.
>>>>>>>> Otherwise it's impossible to predict what's wrong with the
>>>>>>>> system.
>>>>>>>> Check
>>>>>>>> logs and find exceptions and other errornous stuff.
>>>>>>>>
>>>>>>>> 6 нояб. 2017 г. 18:14 пользователь <he...@powershield.ru>
>>>>>>>> написал:
>>>>>>>>
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>>>
>>>>>>>>> ACS: 4.8.0
>>>>>>>>> XenServer 6.5SP1
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> We have got some problem with additional space for our client.
>>>>>>>>>
>>>>>>>>> We are created new Disk Offering, then we tryed to attach for
>>>>>>>>> our client, but got error: Unable to find suitable primary
>>>>>>>>> storage when creating volume
>>>>>>>>>
>>>>>>>>> We checked datadisk and we see, there is not hypervisor (should
>>>>>>>>> it be showed? Now saying: None)
>>>>>>>>>
>>>>>>>>> We tryed to create new instance, but also unlucky.
>>>>>>>>>
>>>>>>>>> After it, we restarted VR and he can't start. He started and
>>>>>>>>> again goes to offline. We tryed to check from XenCenter and got
>>>>>>>>> this reason:
>>>>>>>>> apache2:
>>>>>>>>> Could not reliably determine the server's fully qualified
>>>>>>>>> domain name, using 127.0.0.1 for ServerName.
>>>>>>>>>
>>>>>>>>> We delete VR and now, we can not start it manually and he is
>>>>>>>>> not starting automatically about 40min.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
Re: VR starting problem
Posted by he...@powershield.ru.
Hello,
Here is some of log.
https://pastebin.com/JU0qr69V
Interested line: dataCenterId:: 2 // podId:: null // clusterId:: null
As you can see, datacenter (zone) showing, but can't see pod and
cluster....
Paul Angus писал 2017-11-06 22:44:
> Please could you use something like pastebin (https://pastebin.com/)
> to paste larger sections of your logs.
> The section you have pasted is more likely from XenServer trying to
> remove a disk image that was never created in the first place. It may
> not be the root cause.
>
> Kind regards,
>
> Paul Angus
>
> paul.angus@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> @shapeblue
>
>
>
>
> -----Original Message-----
> From: hello@powershield.ru [mailto:hello@powershield.ru]
> Sent: 06 November 2017 18:14
> To: users@cloudstack.apache.org
> Subject: Re: VR starting problem
>
> We deleted system VM's and also, can't start.
>
> Now seeing the error:
>
>
> INFO [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-11:ctx-eaf4ab20 job-97/job-14993 ctx-7391ff7c)
> (logid:66199d2c) Unable to contact resource.
> com.cloud.exception.StorageUnavailableException: Resource
> [StoragePool:6] is unreachable: Unable to create
> Vol[2034|vm=2010|ROOT]:com.cloud.utils.exception.CloudRuntimeException:
> Catch Exception com.xensource.xenapi.Types$UuidInvalid :VDI getByUuid
> for uuid: 5e2f645f-9e5d-45df-a268-dbe9c6c9eef9 failed due to The uuid
> you supplied was invalid.
>
> 2017-11-06 20:16:49,498 DEBUG [c.c.h.x.r.XenServerStorageProcessor]
> (DirectAgent-6:ctx-b7ab797e) (logid:d3084120) Catch Exception
> com.xensource.xenapi.Types$UuidInvalid :VDI getByUuid for uuid:
> 5e2f645f-9e5d-45df-a268-dbe9c6c9eef9 failed due to The uuid you
> supplied was invalid.
> 2017-11-06 20:16:49,501 WARN [c.c.h.x.r.XenServerStorageProcessor]
> (DirectAgent-6:ctx-b7ab797e) (logid:d3084120) Unable to create volume;
> Pool=volumeTO[uuid=efa30838-f8e4-4186-a299-d2ae716d9081|path=null|datastore=PrimaryDataStoreTO[uuid=SR-UK|name=null|id=6|pooltype=PreSetup]];
> Disk:
> com.cloud.utils.exception.CloudRuntimeException: Catch Exception
> com.xensource.xenapi.Types$UuidInvalid :VDI getByUuid for uuid:
> 5e2f645f-9e5d-45df-a268-dbe9c6c9eef9 failed due to The uuid you
> supplied was invalid.
>
>
>
>
>
> hello@powershield.ru писал 2017-11-06 20:48:
>> Changed few hours ago, and anyway still the same problem.
>>
>>
>>
>> Ivan Kudryavtsev писал 2017-11-06 20:40:
>>> Hi. Don't mix "used" and "allocated". Just change the specified
>>> threshold to 0.98 and go ahead.
>>>
>>> 7 нояб. 2017 г. 0:22 пользователь <he...@powershield.ru> написал:
>>>
>>>> No, i re-checked maybe 10 times. I just added some more space, but
>>>> the problem is same. All VM's working but if client restart - they
>>>> goes down, with the same error with insufficient storage.
>>>>
>>>>
>>>> Ivan Kudryavtsev писал 2017-11-06 20:16:
>>>>
>>>>> Could it be, that your storage, cluster, pod, zone is in the
>>>>> maintanence mode?
>>>>>
>>>>> 6 нояб. 2017 г. 23:51 пользователь "Ivan Kudryavtsev" <
>>>>> kudryavtsev_ia@bw-sw.com> написал:
>>>>>
>>>>> Hi. Try to find in early logs what exactly resource is
>>>>> insufficient.
>>>>> Try
>>>>>> to play with threshold of it.
>>>>>>
>>>>>> 6 нояб. 2017 г. 21:57 пользователь <he...@powershield.ru> написал:
>>>>>>
>>>>>> Now, i see the problem, then VR restarting.
>>>>>>>
>>>>>>> 2017-11-06 11:54:15,821 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>> (API-Job-Executor-43:ctx-02214ed9 job-14455) (logid:18ad84ad)
>>>>>>> Complete async job-14455, jobStatus: FAILED, resultCode: 530,
>>>>>>> result:
>>>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>>>>>>> uuidList":[],"errorcode":530,"errortext":"Resource [Host:17] is
>>>>>>> unreachable: Host 17: Unable to start instance due to Unable to
>>>>>>> start
>>>>>>> VM:18985a14-3cb5-46d5-a3f0-3a065cd8b014 due to error in
>>>>>>> finalizeStart, not retrying"}
>>>>>>> 2017-11-06 15:47:27,814 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>> (API-Job-Executor-69:ctx-be0b5550 job-14663) (logid:8b609324)
>>>>>>> Complete async job-14663, jobStatus: FAILED, resultCode: 530,
>>>>>>> result:
>>>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>>>>>>> uuidList":[],"errorcode":530,"errortext":"Unable to start a VM
>>>>>>> due to insufficient capacity"}
>>>>>>> 2017-11-06 16:54:19,055 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>> (API-Job-Executor-70:ctx-deaaf8e8 job-14668) (logid:6a7700af)
>>>>>>> Complete async job-14668, jobStatus: FAILED, resultCode: 530,
>>>>>>> result:
>>>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>>>>>>> uuidList":[],"errorcode":530,"errortext":"Resource [DataCenter:2]
>>>>>>> is
>>>>>>> unreachable: Unable to reboot domR, it is not in right state
>>>>>>> Stopped"}
>>>>>>> 2017-11-06 16:58:54,235 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>> (API-Job-Executor-71:ctx-b26b3b33 job-14669) (logid:436e6183)
>>>>>>> Complete async job-14669, jobStatus: FAILED, resultCode: 530,
>>>>>>> result:
>>>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>>>>>>> uuidList":[],"errorcode":530,"errortext":"Unable to create a
>>>>>>> deployment for VM[DomainRouter|r-1919-VM]"}
>>>>>>>
>>>>>>> I'm pretty sure, what i have enough resources:
>>>>>>>
>>>>>>> Primary Storage Used
>>>>>>> Allocated: 4.22 TB / 4.88 TB
>>>>>>>
>>>>>>> Primary Storage Allocated
>>>>>>> Allocated: 3.02 TB / 4.88 TB
>>>>>>>
>>>>>>> Memory Allocated
>>>>>>> Allocated: 78.00 GB / 597.93 GB
>>>>>>>
>>>>>>> CPU Allocated
>>>>>>> Allocated: 81.00 GHz / 440.00 GHz
>>>>>>>
>>>>>>> Shared Network IPs
>>>>>>> Allocated: 34 / 253
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Ivan Kudryavtsev писал 2017-11-06 15:28:
>>>>>>>
>>>>>>> Hi, you should dive into management server logs to find relevant
>>>>>>>> messages.
>>>>>>>> Otherwise it's impossible to predict what's wrong with the
>>>>>>>> system.
>>>>>>>> Check
>>>>>>>> logs and find exceptions and other errornous stuff.
>>>>>>>>
>>>>>>>> 6 нояб. 2017 г. 18:14 пользователь <he...@powershield.ru>
>>>>>>>> написал:
>>>>>>>>
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>>>
>>>>>>>>> ACS: 4.8.0
>>>>>>>>> XenServer 6.5SP1
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> We have got some problem with additional space for our client.
>>>>>>>>>
>>>>>>>>> We are created new Disk Offering, then we tryed to attach for
>>>>>>>>> our client, but got error: Unable to find suitable primary
>>>>>>>>> storage when creating volume
>>>>>>>>>
>>>>>>>>> We checked datadisk and we see, there is not hypervisor (should
>>>>>>>>> it be showed? Now saying: None)
>>>>>>>>>
>>>>>>>>> We tryed to create new instance, but also unlucky.
>>>>>>>>>
>>>>>>>>> After it, we restarted VR and he can't start. He started and
>>>>>>>>> again goes to offline. We tryed to check from XenCenter and got
>>>>>>>>> this reason:
>>>>>>>>> apache2:
>>>>>>>>> Could not reliably determine the server's fully qualified
>>>>>>>>> domain name, using 127.0.0.1 for ServerName.
>>>>>>>>>
>>>>>>>>> We delete VR and now, we can not start it manually and he is
>>>>>>>>> not starting automatically about 40min.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
RE: VR starting problem
Posted by Paul Angus <pa...@shapeblue.com>.
Please could you use something like pastebin (https://pastebin.com/) to paste larger sections of your logs.
The section you have pasted is more likely from XenServer trying to remove a disk image that was never created in the first place. It may not be the root cause.
Kind regards,
Paul Angus
paul.angus@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
-----Original Message-----
From: hello@powershield.ru [mailto:hello@powershield.ru]
Sent: 06 November 2017 18:14
To: users@cloudstack.apache.org
Subject: Re: VR starting problem
We deleted system VM's and also, can't start.
Now seeing the error:
INFO [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-11:ctx-eaf4ab20 job-97/job-14993 ctx-7391ff7c)
(logid:66199d2c) Unable to contact resource.
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:6] is unreachable: Unable to create
Vol[2034|vm=2010|ROOT]:com.cloud.utils.exception.CloudRuntimeException:
Catch Exception com.xensource.xenapi.Types$UuidInvalid :VDI getByUuid for uuid: 5e2f645f-9e5d-45df-a268-dbe9c6c9eef9 failed due to The uuid you supplied was invalid.
2017-11-06 20:16:49,498 DEBUG [c.c.h.x.r.XenServerStorageProcessor]
(DirectAgent-6:ctx-b7ab797e) (logid:d3084120) Catch Exception com.xensource.xenapi.Types$UuidInvalid :VDI getByUuid for uuid:
5e2f645f-9e5d-45df-a268-dbe9c6c9eef9 failed due to The uuid you supplied was invalid.
2017-11-06 20:16:49,501 WARN [c.c.h.x.r.XenServerStorageProcessor]
(DirectAgent-6:ctx-b7ab797e) (logid:d3084120) Unable to create volume; Pool=volumeTO[uuid=efa30838-f8e4-4186-a299-d2ae716d9081|path=null|datastore=PrimaryDataStoreTO[uuid=SR-UK|name=null|id=6|pooltype=PreSetup]];
Disk:
com.cloud.utils.exception.CloudRuntimeException: Catch Exception com.xensource.xenapi.Types$UuidInvalid :VDI getByUuid for uuid:
5e2f645f-9e5d-45df-a268-dbe9c6c9eef9 failed due to The uuid you supplied was invalid.
hello@powershield.ru писал 2017-11-06 20:48:
> Changed few hours ago, and anyway still the same problem.
>
>
>
> Ivan Kudryavtsev писал 2017-11-06 20:40:
>> Hi. Don't mix "used" and "allocated". Just change the specified
>> threshold to 0.98 and go ahead.
>>
>> 7 нояб. 2017 г. 0:22 пользователь <he...@powershield.ru> написал:
>>
>>> No, i re-checked maybe 10 times. I just added some more space, but
>>> the problem is same. All VM's working but if client restart - they
>>> goes down, with the same error with insufficient storage.
>>>
>>>
>>> Ivan Kudryavtsev писал 2017-11-06 20:16:
>>>
>>>> Could it be, that your storage, cluster, pod, zone is in the
>>>> maintanence mode?
>>>>
>>>> 6 нояб. 2017 г. 23:51 пользователь "Ivan Kudryavtsev" <
>>>> kudryavtsev_ia@bw-sw.com> написал:
>>>>
>>>> Hi. Try to find in early logs what exactly resource is insufficient.
>>>> Try
>>>>> to play with threshold of it.
>>>>>
>>>>> 6 нояб. 2017 г. 21:57 пользователь <he...@powershield.ru> написал:
>>>>>
>>>>> Now, i see the problem, then VR restarting.
>>>>>>
>>>>>> 2017-11-06 11:54:15,821 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (API-Job-Executor-43:ctx-02214ed9 job-14455) (logid:18ad84ad)
>>>>>> Complete async job-14455, jobStatus: FAILED, resultCode: 530,
>>>>>> result:
>>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>>>>>> uuidList":[],"errorcode":530,"errortext":"Resource [Host:17] is
>>>>>> unreachable: Host 17: Unable to start instance due to Unable to
>>>>>> start
>>>>>> VM:18985a14-3cb5-46d5-a3f0-3a065cd8b014 due to error in
>>>>>> finalizeStart, not retrying"}
>>>>>> 2017-11-06 15:47:27,814 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (API-Job-Executor-69:ctx-be0b5550 job-14663) (logid:8b609324)
>>>>>> Complete async job-14663, jobStatus: FAILED, resultCode: 530,
>>>>>> result:
>>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>>>>>> uuidList":[],"errorcode":530,"errortext":"Unable to start a VM
>>>>>> due to insufficient capacity"}
>>>>>> 2017-11-06 16:54:19,055 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (API-Job-Executor-70:ctx-deaaf8e8 job-14668) (logid:6a7700af)
>>>>>> Complete async job-14668, jobStatus: FAILED, resultCode: 530,
>>>>>> result:
>>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>>>>>> uuidList":[],"errorcode":530,"errortext":"Resource [DataCenter:2]
>>>>>> is
>>>>>> unreachable: Unable to reboot domR, it is not in right state
>>>>>> Stopped"}
>>>>>> 2017-11-06 16:58:54,235 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (API-Job-Executor-71:ctx-b26b3b33 job-14669) (logid:436e6183)
>>>>>> Complete async job-14669, jobStatus: FAILED, resultCode: 530,
>>>>>> result:
>>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>>>>>> uuidList":[],"errorcode":530,"errortext":"Unable to create a
>>>>>> deployment for VM[DomainRouter|r-1919-VM]"}
>>>>>>
>>>>>> I'm pretty sure, what i have enough resources:
>>>>>>
>>>>>> Primary Storage Used
>>>>>> Allocated: 4.22 TB / 4.88 TB
>>>>>>
>>>>>> Primary Storage Allocated
>>>>>> Allocated: 3.02 TB / 4.88 TB
>>>>>>
>>>>>> Memory Allocated
>>>>>> Allocated: 78.00 GB / 597.93 GB
>>>>>>
>>>>>> CPU Allocated
>>>>>> Allocated: 81.00 GHz / 440.00 GHz
>>>>>>
>>>>>> Shared Network IPs
>>>>>> Allocated: 34 / 253
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Ivan Kudryavtsev писал 2017-11-06 15:28:
>>>>>>
>>>>>> Hi, you should dive into management server logs to find relevant
>>>>>>> messages.
>>>>>>> Otherwise it's impossible to predict what's wrong with the
>>>>>>> system.
>>>>>>> Check
>>>>>>> logs and find exceptions and other errornous stuff.
>>>>>>>
>>>>>>> 6 нояб. 2017 г. 18:14 пользователь <he...@powershield.ru>
>>>>>>> написал:
>>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>>>
>>>>>>>> ACS: 4.8.0
>>>>>>>> XenServer 6.5SP1
>>>>>>>>
>>>>>>>>
>>>>>>>> We have got some problem with additional space for our client.
>>>>>>>>
>>>>>>>> We are created new Disk Offering, then we tryed to attach for
>>>>>>>> our client, but got error: Unable to find suitable primary
>>>>>>>> storage when creating volume
>>>>>>>>
>>>>>>>> We checked datadisk and we see, there is not hypervisor (should
>>>>>>>> it be showed? Now saying: None)
>>>>>>>>
>>>>>>>> We tryed to create new instance, but also unlucky.
>>>>>>>>
>>>>>>>> After it, we restarted VR and he can't start. He started and
>>>>>>>> again goes to offline. We tryed to check from XenCenter and got
>>>>>>>> this reason:
>>>>>>>> apache2:
>>>>>>>> Could not reliably determine the server's fully qualified
>>>>>>>> domain name, using 127.0.0.1 for ServerName.
>>>>>>>>
>>>>>>>> We delete VR and now, we can not start it manually and he is
>>>>>>>> not starting automatically about 40min.
>>>>>>>>
>>>>>>>>
>>>>>>>>
Re: VR starting problem
Posted by he...@powershield.ru.
We deleted system VM's and also, can't start.
Now seeing the error:
INFO [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-11:ctx-eaf4ab20 job-97/job-14993 ctx-7391ff7c)
(logid:66199d2c) Unable to contact resource.
com.cloud.exception.StorageUnavailableException: Resource
[StoragePool:6] is unreachable: Unable to create
Vol[2034|vm=2010|ROOT]:com.cloud.utils.exception.CloudRuntimeException:
Catch Exception com.xensource.xenapi.Types$UuidInvalid :VDI getByUuid
for uuid: 5e2f645f-9e5d-45df-a268-dbe9c6c9eef9 failed due to The uuid
you supplied was invalid.
2017-11-06 20:16:49,498 DEBUG [c.c.h.x.r.XenServerStorageProcessor]
(DirectAgent-6:ctx-b7ab797e) (logid:d3084120) Catch Exception
com.xensource.xenapi.Types$UuidInvalid :VDI getByUuid for uuid:
5e2f645f-9e5d-45df-a268-dbe9c6c9eef9 failed due to The uuid you supplied
was invalid.
2017-11-06 20:16:49,501 WARN [c.c.h.x.r.XenServerStorageProcessor]
(DirectAgent-6:ctx-b7ab797e) (logid:d3084120) Unable to create volume;
Pool=volumeTO[uuid=efa30838-f8e4-4186-a299-d2ae716d9081|path=null|datastore=PrimaryDataStoreTO[uuid=SR-UK|name=null|id=6|pooltype=PreSetup]];
Disk:
com.cloud.utils.exception.CloudRuntimeException: Catch Exception
com.xensource.xenapi.Types$UuidInvalid :VDI getByUuid for uuid:
5e2f645f-9e5d-45df-a268-dbe9c6c9eef9 failed due to The uuid you supplied
was invalid.
hello@powershield.ru писал 2017-11-06 20:48:
> Changed few hours ago, and anyway still the same problem.
>
>
>
> Ivan Kudryavtsev писал 2017-11-06 20:40:
>> Hi. Don't mix "used" and "allocated". Just change the specified
>> threshold
>> to 0.98 and go ahead.
>>
>> 7 нояб. 2017 г. 0:22 пользователь <he...@powershield.ru> написал:
>>
>>> No, i re-checked maybe 10 times. I just added some more space, but
>>> the
>>> problem is same. All VM's working but if client restart - they goes
>>> down,
>>> with the same error with insufficient storage.
>>>
>>>
>>> Ivan Kudryavtsev писал 2017-11-06 20:16:
>>>
>>>> Could it be, that your storage, cluster, pod, zone is in the
>>>> maintanence
>>>> mode?
>>>>
>>>> 6 нояб. 2017 г. 23:51 пользователь "Ivan Kudryavtsev" <
>>>> kudryavtsev_ia@bw-sw.com> написал:
>>>>
>>>> Hi. Try to find in early logs what exactly resource is insufficient.
>>>> Try
>>>>> to play with threshold of it.
>>>>>
>>>>> 6 нояб. 2017 г. 21:57 пользователь <he...@powershield.ru> написал:
>>>>>
>>>>> Now, i see the problem, then VR restarting.
>>>>>>
>>>>>> 2017-11-06 11:54:15,821 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (API-Job-Executor-43:ctx-02214ed9 job-14455) (logid:18ad84ad)
>>>>>> Complete
>>>>>> async job-14455, jobStatus: FAILED, resultCode: 530, result:
>>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>>>>>> uuidList":[],"errorcode":530,"errortext":"Resource [Host:17] is
>>>>>> unreachable: Host 17: Unable to start instance due to Unable to
>>>>>> start
>>>>>> VM:18985a14-3cb5-46d5-a3f0-3a065cd8b014 due to error in
>>>>>> finalizeStart,
>>>>>> not retrying"}
>>>>>> 2017-11-06 15:47:27,814 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (API-Job-Executor-69:ctx-be0b5550 job-14663) (logid:8b609324)
>>>>>> Complete
>>>>>> async job-14663, jobStatus: FAILED, resultCode: 530, result:
>>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>>>>>> uuidList":[],"errorcode":530,"errortext":"Unable to start a VM due
>>>>>> to
>>>>>> insufficient capacity"}
>>>>>> 2017-11-06 16:54:19,055 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (API-Job-Executor-70:ctx-deaaf8e8 job-14668) (logid:6a7700af)
>>>>>> Complete
>>>>>> async job-14668, jobStatus: FAILED, resultCode: 530, result:
>>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>>>>>> uuidList":[],"errorcode":530,"errortext":"Resource [DataCenter:2]
>>>>>> is
>>>>>> unreachable: Unable to reboot domR, it is not in right state
>>>>>> Stopped"}
>>>>>> 2017-11-06 16:58:54,235 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (API-Job-Executor-71:ctx-b26b3b33 job-14669) (logid:436e6183)
>>>>>> Complete
>>>>>> async job-14669, jobStatus: FAILED, resultCode: 530, result:
>>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>>>>>> uuidList":[],"errorcode":530,"errortext":"Unable to create a
>>>>>> deployment
>>>>>> for VM[DomainRouter|r-1919-VM]"}
>>>>>>
>>>>>> I'm pretty sure, what i have enough resources:
>>>>>>
>>>>>> Primary Storage Used
>>>>>> Allocated: 4.22 TB / 4.88 TB
>>>>>>
>>>>>> Primary Storage Allocated
>>>>>> Allocated: 3.02 TB / 4.88 TB
>>>>>>
>>>>>> Memory Allocated
>>>>>> Allocated: 78.00 GB / 597.93 GB
>>>>>>
>>>>>> CPU Allocated
>>>>>> Allocated: 81.00 GHz / 440.00 GHz
>>>>>>
>>>>>> Shared Network IPs
>>>>>> Allocated: 34 / 253
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Ivan Kudryavtsev писал 2017-11-06 15:28:
>>>>>>
>>>>>> Hi, you should dive into management server logs to find relevant
>>>>>>> messages.
>>>>>>> Otherwise it's impossible to predict what's wrong with the
>>>>>>> system.
>>>>>>> Check
>>>>>>> logs and find exceptions and other errornous stuff.
>>>>>>>
>>>>>>> 6 нояб. 2017 г. 18:14 пользователь <he...@powershield.ru>
>>>>>>> написал:
>>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>>>
>>>>>>>> ACS: 4.8.0
>>>>>>>> XenServer 6.5SP1
>>>>>>>>
>>>>>>>>
>>>>>>>> We have got some problem with additional space for our client.
>>>>>>>>
>>>>>>>> We are created new Disk Offering, then we tryed to attach for
>>>>>>>> our
>>>>>>>> client,
>>>>>>>> but got error: Unable to find suitable primary storage when
>>>>>>>> creating
>>>>>>>> volume
>>>>>>>>
>>>>>>>> We checked datadisk and we see, there is not hypervisor (should
>>>>>>>> it be
>>>>>>>> showed? Now saying: None)
>>>>>>>>
>>>>>>>> We tryed to create new instance, but also unlucky.
>>>>>>>>
>>>>>>>> After it, we restarted VR and he can't start. He started and
>>>>>>>> again
>>>>>>>> goes
>>>>>>>> to
>>>>>>>> offline. We tryed to check from XenCenter and got this reason:
>>>>>>>> apache2:
>>>>>>>> Could not reliably determine the server's fully qualified domain
>>>>>>>> name,
>>>>>>>> using 127.0.0.1 for ServerName.
>>>>>>>>
>>>>>>>> We delete VR and now, we can not start it manually and he is not
>>>>>>>> starting
>>>>>>>> automatically about 40min.
>>>>>>>>
>>>>>>>>
>>>>>>>>
Re: VR starting problem
Posted by he...@powershield.ru.
Changed few hours ago, and anyway still the same problem.
Ivan Kudryavtsev писал 2017-11-06 20:40:
> Hi. Don't mix "used" and "allocated". Just change the specified
> threshold
> to 0.98 and go ahead.
>
> 7 нояб. 2017 г. 0:22 пользователь <he...@powershield.ru> написал:
>
>> No, i re-checked maybe 10 times. I just added some more space, but the
>> problem is same. All VM's working but if client restart - they goes
>> down,
>> with the same error with insufficient storage.
>>
>>
>> Ivan Kudryavtsev писал 2017-11-06 20:16:
>>
>>> Could it be, that your storage, cluster, pod, zone is in the
>>> maintanence
>>> mode?
>>>
>>> 6 нояб. 2017 г. 23:51 пользователь "Ivan Kudryavtsev" <
>>> kudryavtsev_ia@bw-sw.com> написал:
>>>
>>> Hi. Try to find in early logs what exactly resource is insufficient.
>>> Try
>>>> to play with threshold of it.
>>>>
>>>> 6 нояб. 2017 г. 21:57 пользователь <he...@powershield.ru> написал:
>>>>
>>>> Now, i see the problem, then VR restarting.
>>>>>
>>>>> 2017-11-06 11:54:15,821 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>> (API-Job-Executor-43:ctx-02214ed9 job-14455) (logid:18ad84ad)
>>>>> Complete
>>>>> async job-14455, jobStatus: FAILED, resultCode: 530, result:
>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>>>>> uuidList":[],"errorcode":530,"errortext":"Resource [Host:17] is
>>>>> unreachable: Host 17: Unable to start instance due to Unable to
>>>>> start
>>>>> VM:18985a14-3cb5-46d5-a3f0-3a065cd8b014 due to error in
>>>>> finalizeStart,
>>>>> not retrying"}
>>>>> 2017-11-06 15:47:27,814 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>> (API-Job-Executor-69:ctx-be0b5550 job-14663) (logid:8b609324)
>>>>> Complete
>>>>> async job-14663, jobStatus: FAILED, resultCode: 530, result:
>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>>>>> uuidList":[],"errorcode":530,"errortext":"Unable to start a VM due
>>>>> to
>>>>> insufficient capacity"}
>>>>> 2017-11-06 16:54:19,055 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>> (API-Job-Executor-70:ctx-deaaf8e8 job-14668) (logid:6a7700af)
>>>>> Complete
>>>>> async job-14668, jobStatus: FAILED, resultCode: 530, result:
>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>>>>> uuidList":[],"errorcode":530,"errortext":"Resource [DataCenter:2]
>>>>> is
>>>>> unreachable: Unable to reboot domR, it is not in right state
>>>>> Stopped"}
>>>>> 2017-11-06 16:58:54,235 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>> (API-Job-Executor-71:ctx-b26b3b33 job-14669) (logid:436e6183)
>>>>> Complete
>>>>> async job-14669, jobStatus: FAILED, resultCode: 530, result:
>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>>>>> uuidList":[],"errorcode":530,"errortext":"Unable to create a
>>>>> deployment
>>>>> for VM[DomainRouter|r-1919-VM]"}
>>>>>
>>>>> I'm pretty sure, what i have enough resources:
>>>>>
>>>>> Primary Storage Used
>>>>> Allocated: 4.22 TB / 4.88 TB
>>>>>
>>>>> Primary Storage Allocated
>>>>> Allocated: 3.02 TB / 4.88 TB
>>>>>
>>>>> Memory Allocated
>>>>> Allocated: 78.00 GB / 597.93 GB
>>>>>
>>>>> CPU Allocated
>>>>> Allocated: 81.00 GHz / 440.00 GHz
>>>>>
>>>>> Shared Network IPs
>>>>> Allocated: 34 / 253
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Ivan Kudryavtsev писал 2017-11-06 15:28:
>>>>>
>>>>> Hi, you should dive into management server logs to find relevant
>>>>>> messages.
>>>>>> Otherwise it's impossible to predict what's wrong with the system.
>>>>>> Check
>>>>>> logs and find exceptions and other errornous stuff.
>>>>>>
>>>>>> 6 нояб. 2017 г. 18:14 пользователь <he...@powershield.ru> написал:
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>>>
>>>>>>> ACS: 4.8.0
>>>>>>> XenServer 6.5SP1
>>>>>>>
>>>>>>>
>>>>>>> We have got some problem with additional space for our client.
>>>>>>>
>>>>>>> We are created new Disk Offering, then we tryed to attach for our
>>>>>>> client,
>>>>>>> but got error: Unable to find suitable primary storage when
>>>>>>> creating
>>>>>>> volume
>>>>>>>
>>>>>>> We checked datadisk and we see, there is not hypervisor (should
>>>>>>> it be
>>>>>>> showed? Now saying: None)
>>>>>>>
>>>>>>> We tryed to create new instance, but also unlucky.
>>>>>>>
>>>>>>> After it, we restarted VR and he can't start. He started and
>>>>>>> again
>>>>>>> goes
>>>>>>> to
>>>>>>> offline. We tryed to check from XenCenter and got this reason:
>>>>>>> apache2:
>>>>>>> Could not reliably determine the server's fully qualified domain
>>>>>>> name,
>>>>>>> using 127.0.0.1 for ServerName.
>>>>>>>
>>>>>>> We delete VR and now, we can not start it manually and he is not
>>>>>>> starting
>>>>>>> automatically about 40min.
>>>>>>>
>>>>>>>
>>>>>>>
Re: VR starting problem
Posted by Ivan Kudryavtsev <ku...@bw-sw.com>.
Hi. Don't mix "used" and "allocated". Just change the specified threshold
to 0.98 and go ahead.
7 нояб. 2017 г. 0:22 пользователь <he...@powershield.ru> написал:
> No, i re-checked maybe 10 times. I just added some more space, but the
> problem is same. All VM's working but if client restart - they goes down,
> with the same error with insufficient storage.
>
>
> Ivan Kudryavtsev писал 2017-11-06 20:16:
>
>> Could it be, that your storage, cluster, pod, zone is in the maintanence
>> mode?
>>
>> 6 нояб. 2017 г. 23:51 пользователь "Ivan Kudryavtsev" <
>> kudryavtsev_ia@bw-sw.com> написал:
>>
>> Hi. Try to find in early logs what exactly resource is insufficient. Try
>>> to play with threshold of it.
>>>
>>> 6 нояб. 2017 г. 21:57 пользователь <he...@powershield.ru> написал:
>>>
>>> Now, i see the problem, then VR restarting.
>>>>
>>>> 2017-11-06 11:54:15,821 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-43:ctx-02214ed9 job-14455) (logid:18ad84ad) Complete
>>>> async job-14455, jobStatus: FAILED, resultCode: 530, result:
>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>>>> uuidList":[],"errorcode":530,"errortext":"Resource [Host:17] is
>>>> unreachable: Host 17: Unable to start instance due to Unable to start
>>>> VM:18985a14-3cb5-46d5-a3f0-3a065cd8b014 due to error in finalizeStart,
>>>> not retrying"}
>>>> 2017-11-06 15:47:27,814 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-69:ctx-be0b5550 job-14663) (logid:8b609324) Complete
>>>> async job-14663, jobStatus: FAILED, resultCode: 530, result:
>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>>>> uuidList":[],"errorcode":530,"errortext":"Unable to start a VM due to
>>>> insufficient capacity"}
>>>> 2017-11-06 16:54:19,055 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-70:ctx-deaaf8e8 job-14668) (logid:6a7700af) Complete
>>>> async job-14668, jobStatus: FAILED, resultCode: 530, result:
>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>>>> uuidList":[],"errorcode":530,"errortext":"Resource [DataCenter:2] is
>>>> unreachable: Unable to reboot domR, it is not in right state Stopped"}
>>>> 2017-11-06 16:58:54,235 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-71:ctx-b26b3b33 job-14669) (logid:436e6183) Complete
>>>> async job-14669, jobStatus: FAILED, resultCode: 530, result:
>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>>>> uuidList":[],"errorcode":530,"errortext":"Unable to create a deployment
>>>> for VM[DomainRouter|r-1919-VM]"}
>>>>
>>>> I'm pretty sure, what i have enough resources:
>>>>
>>>> Primary Storage Used
>>>> Allocated: 4.22 TB / 4.88 TB
>>>>
>>>> Primary Storage Allocated
>>>> Allocated: 3.02 TB / 4.88 TB
>>>>
>>>> Memory Allocated
>>>> Allocated: 78.00 GB / 597.93 GB
>>>>
>>>> CPU Allocated
>>>> Allocated: 81.00 GHz / 440.00 GHz
>>>>
>>>> Shared Network IPs
>>>> Allocated: 34 / 253
>>>>
>>>>
>>>>
>>>>
>>>> Ivan Kudryavtsev писал 2017-11-06 15:28:
>>>>
>>>> Hi, you should dive into management server logs to find relevant
>>>>> messages.
>>>>> Otherwise it's impossible to predict what's wrong with the system.
>>>>> Check
>>>>> logs and find exceptions and other errornous stuff.
>>>>>
>>>>> 6 нояб. 2017 г. 18:14 пользователь <he...@powershield.ru> написал:
>>>>>
>>>>> Hello,
>>>>>
>>>>>>
>>>>>> ACS: 4.8.0
>>>>>> XenServer 6.5SP1
>>>>>>
>>>>>>
>>>>>> We have got some problem with additional space for our client.
>>>>>>
>>>>>> We are created new Disk Offering, then we tryed to attach for our
>>>>>> client,
>>>>>> but got error: Unable to find suitable primary storage when creating
>>>>>> volume
>>>>>>
>>>>>> We checked datadisk and we see, there is not hypervisor (should it be
>>>>>> showed? Now saying: None)
>>>>>>
>>>>>> We tryed to create new instance, but also unlucky.
>>>>>>
>>>>>> After it, we restarted VR and he can't start. He started and again
>>>>>> goes
>>>>>> to
>>>>>> offline. We tryed to check from XenCenter and got this reason:
>>>>>> apache2:
>>>>>> Could not reliably determine the server's fully qualified domain name,
>>>>>> using 127.0.0.1 for ServerName.
>>>>>>
>>>>>> We delete VR and now, we can not start it manually and he is not
>>>>>> starting
>>>>>> automatically about 40min.
>>>>>>
>>>>>>
>>>>>>
Re: VR starting problem
Posted by he...@powershield.ru.
No, i re-checked maybe 10 times. I just added some more space, but the
problem is same. All VM's working but if client restart - they goes
down, with the same error with insufficient storage.
Ivan Kudryavtsev писал 2017-11-06 20:16:
> Could it be, that your storage, cluster, pod, zone is in the
> maintanence
> mode?
>
> 6 нояб. 2017 г. 23:51 пользователь "Ivan Kudryavtsev" <
> kudryavtsev_ia@bw-sw.com> написал:
>
>> Hi. Try to find in early logs what exactly resource is insufficient.
>> Try
>> to play with threshold of it.
>>
>> 6 нояб. 2017 г. 21:57 пользователь <he...@powershield.ru> написал:
>>
>>> Now, i see the problem, then VR restarting.
>>>
>>> 2017-11-06 11:54:15,821 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> (API-Job-Executor-43:ctx-02214ed9 job-14455) (logid:18ad84ad)
>>> Complete
>>> async job-14455, jobStatus: FAILED, resultCode: 530, result:
>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>>> uuidList":[],"errorcode":530,"errortext":"Resource [Host:17] is
>>> unreachable: Host 17: Unable to start instance due to Unable to start
>>> VM:18985a14-3cb5-46d5-a3f0-3a065cd8b014 due to error in
>>> finalizeStart,
>>> not retrying"}
>>> 2017-11-06 15:47:27,814 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> (API-Job-Executor-69:ctx-be0b5550 job-14663) (logid:8b609324)
>>> Complete
>>> async job-14663, jobStatus: FAILED, resultCode: 530, result:
>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>>> uuidList":[],"errorcode":530,"errortext":"Unable to start a VM due to
>>> insufficient capacity"}
>>> 2017-11-06 16:54:19,055 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> (API-Job-Executor-70:ctx-deaaf8e8 job-14668) (logid:6a7700af)
>>> Complete
>>> async job-14668, jobStatus: FAILED, resultCode: 530, result:
>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>>> uuidList":[],"errorcode":530,"errortext":"Resource [DataCenter:2] is
>>> unreachable: Unable to reboot domR, it is not in right state
>>> Stopped"}
>>> 2017-11-06 16:58:54,235 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> (API-Job-Executor-71:ctx-b26b3b33 job-14669) (logid:436e6183)
>>> Complete
>>> async job-14669, jobStatus: FAILED, resultCode: 530, result:
>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>>> uuidList":[],"errorcode":530,"errortext":"Unable to create a
>>> deployment
>>> for VM[DomainRouter|r-1919-VM]"}
>>>
>>> I'm pretty sure, what i have enough resources:
>>>
>>> Primary Storage Used
>>> Allocated: 4.22 TB / 4.88 TB
>>>
>>> Primary Storage Allocated
>>> Allocated: 3.02 TB / 4.88 TB
>>>
>>> Memory Allocated
>>> Allocated: 78.00 GB / 597.93 GB
>>>
>>> CPU Allocated
>>> Allocated: 81.00 GHz / 440.00 GHz
>>>
>>> Shared Network IPs
>>> Allocated: 34 / 253
>>>
>>>
>>>
>>>
>>> Ivan Kudryavtsev писал 2017-11-06 15:28:
>>>
>>>> Hi, you should dive into management server logs to find relevant
>>>> messages.
>>>> Otherwise it's impossible to predict what's wrong with the system.
>>>> Check
>>>> logs and find exceptions and other errornous stuff.
>>>>
>>>> 6 нояб. 2017 г. 18:14 пользователь <he...@powershield.ru> написал:
>>>>
>>>> Hello,
>>>>>
>>>>> ACS: 4.8.0
>>>>> XenServer 6.5SP1
>>>>>
>>>>>
>>>>> We have got some problem with additional space for our client.
>>>>>
>>>>> We are created new Disk Offering, then we tryed to attach for our
>>>>> client,
>>>>> but got error: Unable to find suitable primary storage when
>>>>> creating
>>>>> volume
>>>>>
>>>>> We checked datadisk and we see, there is not hypervisor (should it
>>>>> be
>>>>> showed? Now saying: None)
>>>>>
>>>>> We tryed to create new instance, but also unlucky.
>>>>>
>>>>> After it, we restarted VR and he can't start. He started and again
>>>>> goes
>>>>> to
>>>>> offline. We tryed to check from XenCenter and got this reason:
>>>>> apache2:
>>>>> Could not reliably determine the server's fully qualified domain
>>>>> name,
>>>>> using 127.0.0.1 for ServerName.
>>>>>
>>>>> We delete VR and now, we can not start it manually and he is not
>>>>> starting
>>>>> automatically about 40min.
>>>>>
>>>>>
Re: VR starting problem
Posted by Ivan Kudryavtsev <ku...@bw-sw.com>.
Could it be, that your storage, cluster, pod, zone is in the maintanence
mode?
6 нояб. 2017 г. 23:51 пользователь "Ivan Kudryavtsev" <
kudryavtsev_ia@bw-sw.com> написал:
> Hi. Try to find in early logs what exactly resource is insufficient. Try
> to play with threshold of it.
>
> 6 нояб. 2017 г. 21:57 пользователь <he...@powershield.ru> написал:
>
>> Now, i see the problem, then VR restarting.
>>
>> 2017-11-06 11:54:15,821 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-43:ctx-02214ed9 job-14455) (logid:18ad84ad) Complete
>> async job-14455, jobStatus: FAILED, resultCode: 530, result:
>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>> uuidList":[],"errorcode":530,"errortext":"Resource [Host:17] is
>> unreachable: Host 17: Unable to start instance due to Unable to start
>> VM:18985a14-3cb5-46d5-a3f0-3a065cd8b014 due to error in finalizeStart,
>> not retrying"}
>> 2017-11-06 15:47:27,814 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-69:ctx-be0b5550 job-14663) (logid:8b609324) Complete
>> async job-14663, jobStatus: FAILED, resultCode: 530, result:
>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>> uuidList":[],"errorcode":530,"errortext":"Unable to start a VM due to
>> insufficient capacity"}
>> 2017-11-06 16:54:19,055 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-70:ctx-deaaf8e8 job-14668) (logid:6a7700af) Complete
>> async job-14668, jobStatus: FAILED, resultCode: 530, result:
>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>> uuidList":[],"errorcode":530,"errortext":"Resource [DataCenter:2] is
>> unreachable: Unable to reboot domR, it is not in right state Stopped"}
>> 2017-11-06 16:58:54,235 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-71:ctx-b26b3b33 job-14669) (logid:436e6183) Complete
>> async job-14669, jobStatus: FAILED, resultCode: 530, result:
>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
>> uuidList":[],"errorcode":530,"errortext":"Unable to create a deployment
>> for VM[DomainRouter|r-1919-VM]"}
>>
>> I'm pretty sure, what i have enough resources:
>>
>> Primary Storage Used
>> Allocated: 4.22 TB / 4.88 TB
>>
>> Primary Storage Allocated
>> Allocated: 3.02 TB / 4.88 TB
>>
>> Memory Allocated
>> Allocated: 78.00 GB / 597.93 GB
>>
>> CPU Allocated
>> Allocated: 81.00 GHz / 440.00 GHz
>>
>> Shared Network IPs
>> Allocated: 34 / 253
>>
>>
>>
>>
>> Ivan Kudryavtsev писал 2017-11-06 15:28:
>>
>>> Hi, you should dive into management server logs to find relevant
>>> messages.
>>> Otherwise it's impossible to predict what's wrong with the system. Check
>>> logs and find exceptions and other errornous stuff.
>>>
>>> 6 нояб. 2017 г. 18:14 пользователь <he...@powershield.ru> написал:
>>>
>>> Hello,
>>>>
>>>> ACS: 4.8.0
>>>> XenServer 6.5SP1
>>>>
>>>>
>>>> We have got some problem with additional space for our client.
>>>>
>>>> We are created new Disk Offering, then we tryed to attach for our
>>>> client,
>>>> but got error: Unable to find suitable primary storage when creating
>>>> volume
>>>>
>>>> We checked datadisk and we see, there is not hypervisor (should it be
>>>> showed? Now saying: None)
>>>>
>>>> We tryed to create new instance, but also unlucky.
>>>>
>>>> After it, we restarted VR and he can't start. He started and again goes
>>>> to
>>>> offline. We tryed to check from XenCenter and got this reason: apache2:
>>>> Could not reliably determine the server's fully qualified domain name,
>>>> using 127.0.0.1 for ServerName.
>>>>
>>>> We delete VR and now, we can not start it manually and he is not
>>>> starting
>>>> automatically about 40min.
>>>>
>>>>
Re: VR starting problem
Posted by Ivan Kudryavtsev <ku...@bw-sw.com>.
Hi. Try to find in early logs what exactly resource is insufficient. Try to
play with threshold of it.
6 нояб. 2017 г. 21:57 пользователь <he...@powershield.ru> написал:
> Now, i see the problem, then VR restarting.
>
> 2017-11-06 11:54:15,821 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-43:ctx-02214ed9 job-14455) (logid:18ad84ad) Complete
> async job-14455, jobStatus: FAILED, resultCode: 530, result:
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
> uuidList":[],"errorcode":530,"errortext":"Resource [Host:17] is
> unreachable: Host 17: Unable to start instance due to Unable to start
> VM:18985a14-3cb5-46d5-a3f0-3a065cd8b014 due to error in finalizeStart,
> not retrying"}
> 2017-11-06 15:47:27,814 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-69:ctx-be0b5550 job-14663) (logid:8b609324) Complete
> async job-14663, jobStatus: FAILED, resultCode: 530, result:
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
> uuidList":[],"errorcode":530,"errortext":"Unable to start a VM due to
> insufficient capacity"}
> 2017-11-06 16:54:19,055 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-70:ctx-deaaf8e8 job-14668) (logid:6a7700af) Complete
> async job-14668, jobStatus: FAILED, resultCode: 530, result:
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
> uuidList":[],"errorcode":530,"errortext":"Resource [DataCenter:2] is
> unreachable: Unable to reboot domR, it is not in right state Stopped"}
> 2017-11-06 16:58:54,235 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-71:ctx-b26b3b33 job-14669) (logid:436e6183) Complete
> async job-14669, jobStatus: FAILED, resultCode: 530, result:
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
> uuidList":[],"errorcode":530,"errortext":"Unable to create a deployment
> for VM[DomainRouter|r-1919-VM]"}
>
> I'm pretty sure, what i have enough resources:
>
> Primary Storage Used
> Allocated: 4.22 TB / 4.88 TB
>
> Primary Storage Allocated
> Allocated: 3.02 TB / 4.88 TB
>
> Memory Allocated
> Allocated: 78.00 GB / 597.93 GB
>
> CPU Allocated
> Allocated: 81.00 GHz / 440.00 GHz
>
> Shared Network IPs
> Allocated: 34 / 253
>
>
>
>
> Ivan Kudryavtsev писал 2017-11-06 15:28:
>
>> Hi, you should dive into management server logs to find relevant messages.
>> Otherwise it's impossible to predict what's wrong with the system. Check
>> logs and find exceptions and other errornous stuff.
>>
>> 6 нояб. 2017 г. 18:14 пользователь <he...@powershield.ru> написал:
>>
>> Hello,
>>>
>>> ACS: 4.8.0
>>> XenServer 6.5SP1
>>>
>>>
>>> We have got some problem with additional space for our client.
>>>
>>> We are created new Disk Offering, then we tryed to attach for our client,
>>> but got error: Unable to find suitable primary storage when creating
>>> volume
>>>
>>> We checked datadisk and we see, there is not hypervisor (should it be
>>> showed? Now saying: None)
>>>
>>> We tryed to create new instance, but also unlucky.
>>>
>>> After it, we restarted VR and he can't start. He started and again goes
>>> to
>>> offline. We tryed to check from XenCenter and got this reason: apache2:
>>> Could not reliably determine the server's fully qualified domain name,
>>> using 127.0.0.1 for ServerName.
>>>
>>> We delete VR and now, we can not start it manually and he is not starting
>>> automatically about 40min.
>>>
>>>
Re: VR starting problem
Posted by he...@powershield.ru.
Now, i see the problem, then VR restarting.
2017-11-06 11:54:15,821 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-43:ctx-02214ed9 job-14455) (logid:18ad84ad) Complete
async job-14455, jobStatus: FAILED, resultCode: 530, result:
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Resource
[Host:17] is unreachable: Host 17: Unable to start instance due to
Unable to start VM:18985a14-3cb5-46d5-a3f0-3a065cd8b014 due to error in
finalizeStart, not retrying"}
2017-11-06 15:47:27,814 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-69:ctx-be0b5550 job-14663) (logid:8b609324) Complete
async job-14663, jobStatus: FAILED, resultCode: 530, result:
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Unable
to start a VM due to insufficient capacity"}
2017-11-06 16:54:19,055 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-70:ctx-deaaf8e8 job-14668) (logid:6a7700af) Complete
async job-14668, jobStatus: FAILED, resultCode: 530, result:
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Resource
[DataCenter:2] is unreachable: Unable to reboot domR, it is not in right
state Stopped"}
2017-11-06 16:58:54,235 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-71:ctx-b26b3b33 job-14669) (logid:436e6183) Complete
async job-14669, jobStatus: FAILED, resultCode: 530, result:
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Unable
to create a deployment for VM[DomainRouter|r-1919-VM]"}
I'm pretty sure, what i have enough resources:
Primary Storage Used
Allocated: 4.22 TB / 4.88 TB
Primary Storage Allocated
Allocated: 3.02 TB / 4.88 TB
Memory Allocated
Allocated: 78.00 GB / 597.93 GB
CPU Allocated
Allocated: 81.00 GHz / 440.00 GHz
Shared Network IPs
Allocated: 34 / 253
Ivan Kudryavtsev писал 2017-11-06 15:28:
> Hi, you should dive into management server logs to find relevant
> messages.
> Otherwise it's impossible to predict what's wrong with the system.
> Check
> logs and find exceptions and other errornous stuff.
>
> 6 нояб. 2017 г. 18:14 пользователь <he...@powershield.ru> написал:
>
>> Hello,
>>
>> ACS: 4.8.0
>> XenServer 6.5SP1
>>
>>
>> We have got some problem with additional space for our client.
>>
>> We are created new Disk Offering, then we tryed to attach for our
>> client,
>> but got error: Unable to find suitable primary storage when creating
>> volume
>>
>> We checked datadisk and we see, there is not hypervisor (should it be
>> showed? Now saying: None)
>>
>> We tryed to create new instance, but also unlucky.
>>
>> After it, we restarted VR and he can't start. He started and again
>> goes to
>> offline. We tryed to check from XenCenter and got this reason:
>> apache2:
>> Could not reliably determine the server's fully qualified domain name,
>> using 127.0.0.1 for ServerName.
>>
>> We delete VR and now, we can not start it manually and he is not
>> starting
>> automatically about 40min.
>>
Re: VR starting problem
Posted by Ivan Kudryavtsev <ku...@bw-sw.com>.
Hi, you should dive into management server logs to find relevant messages.
Otherwise it's impossible to predict what's wrong with the system. Check
logs and find exceptions and other errornous stuff.
6 нояб. 2017 г. 18:14 пользователь <he...@powershield.ru> написал:
> Hello,
>
> ACS: 4.8.0
> XenServer 6.5SP1
>
>
> We have got some problem with additional space for our client.
>
> We are created new Disk Offering, then we tryed to attach for our client,
> but got error: Unable to find suitable primary storage when creating volume
>
> We checked datadisk and we see, there is not hypervisor (should it be
> showed? Now saying: None)
>
> We tryed to create new instance, but also unlucky.
>
> After it, we restarted VR and he can't start. He started and again goes to
> offline. We tryed to check from XenCenter and got this reason: apache2:
> Could not reliably determine the server's fully qualified domain name,
> using 127.0.0.1 for ServerName.
>
> We delete VR and now, we can not start it manually and he is not starting
> automatically about 40min.
>