You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Granwille Strauss <gr...@namhost.com.INVALID> on 2022/10/17 06:51:55 UTC
Recurring Snaps Failing
Hi Guys
I enabled recurring snap shots for my Linux VMs and they all seem to be
failing with the following error:
> root@hostname ~ $ grep error
> /var/log/cloudstack/management/management-server.log
> 2022-10-17 03:01:29,474 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd]
> (API-Job-Executor-27:ctx-99d7d702 job-796 ctx-f26a81fe)
> (logid:f9a61b00) Failed to create snapshot due to an internal error
> creating snapshot for volume 635fd5b5-bf14-4cf4-979c-be8a649d5189
> 2022-10-17 03:01:29,476 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-27:ctx-99d7d702 job-796) (logid:f9a61b00) Complete
> async job-796, jobStatus: FAILED, resultCode: 530, result:
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed
> to create snapshot due to an internal error creating snapshot for
> volume 635fd5b5-bf14-4cf4-979c-be8a649d5189"}
> 2022-10-17 03:01:29,478 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd]
> (API-Job-Executor-26:ctx-c7865967 job-794 ctx-0700afb0)
> (logid:5541113c) Failed to create snapshot due to an internal error
> creating snapshot for volume 70ffef79-3aad-43ab-93d1-153fcd34ad15
> 2022-10-17 03:01:29,480 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-26:ctx-c7865967 job-794) (logid:5541113c) Complete
> async job-794, jobStatus: FAILED, resultCode: 530, result:
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed
> to create snapshot due to an internal error creating snapshot for
> volume 70ffef79-3aad-43ab-93d1-153fcd34ad15"}
All I am getting out of this is internal error and nothing else. I have
checked other logs and I am not quite sure where else to look.
Any assistance in this regard will be appreciated.
--
Regards / Groete
<https://www.namhost.com> Granwille Strauss // Senior Systems Admin
*e:* granwille@namhost.com
*m:* +264 81 323 1260 <tel:+264813231260>
*w:* www.namhost.com <https://www.namhost.com/>
<https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
Namhost Internet Services (Pty) Ltd,
24 Black Eagle Rd, Hermanus, 7210, RSA
The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It
is forbidden to copy, forward, or in any way reveal the contents of this
message to anyone without our explicit consent. The integrity and
security of this email cannot be guaranteed over the Internet.
Therefore, the sender will not be held liable for any damage caused by
the message. For our full privacy policy and disclaimers, please go to
https://www.namhost.com/privacy-policy
Powered by AdSigner
<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
Re: Recurring Snaps Failing
Posted by Granwille Strauss <gr...@namhost.com.INVALID>.
Hi Guys
Just another update. I added 1 Gbps port speeds at both my KVM node and
MGMNT node where the storage drives are. This has help significantly
lower the time execution limit that was previously set to 3600 seconds.
I can now work with this limit. However, should VMs increase in size I
will run back into this error.
The problem that I now experience is that the transfer speeds are
limited to 100 Mbps, but I have 1 Gbps in place. I spoke with the ISP
and they said the following:
> Our network engineers have advised that "both interfaces for xxxxx and
> xxxxxx are configured for 1Gbs operation. We does not rate-limit
> client traffic in the data-centre except through port-based general
> bandwidth limiters.
>
> The client must do iperf tests in both directions. To do a reverse
> direction test, add the '-R' parameter to the client side command.
>
> If the iperf tests indicate 1Gbs in both directions, the client should
> examine the copy process more closely.
> What protocol is involved?
> Is there some inherent speed limit to the protocol?
> Is there parameters related to the copy daemon or client which could
> influence the through-put."
Could you kindly confirm what protocols are involved during volume
snapshot process? Are there any speed limits in the global
configurations that we need to look at? Any feedback on this matter will
be appreciated.
On 10/21/22 09:35, Granwille Strauss wrote:
>
> For anyone following this thread, because my KVM server and Cloud
> Management server is connected via public network, the speeds were too
> slow to complete the volume snapshots in time and caused a time out at
> 3600 seconds. As a solution, we increased
> `job.cancel.threshold.minutes` in the global settings from the default
> 60 minutes to 120 minute, which seems to have solved the issue.
>
> However, the proper solution is to implement a good LAN network
> between the two servers instead.
>
> On 10/18/22 17:51, Granwille Strauss wrote:
>>
>> Hi Guys
>>
>> I have logged the issue on GITHub:
>> https://github.com/apache/cloudstack/issues/6836 I would appreciate
>> it if you can take some time to comment on the issue on GITHub, to
>> help better explain what the problem is, please. Thank you very much
>> for your assistance and patience thus far. Hopefully we can get a
>> solution soon.
>>
>> On 10/18/22 17:04, Granwille Strauss wrote:
>>>
>>> Hi
>>>
>>> It failed again right after 1 hour, even after I changed the 'wait'
>>> from 1800 to 3600. Is there not another "wait" setting that may need
>>> to be increased?
>>>
>>> On 10/18/22 15:38, Granwille Strauss wrote:
>>>>
>>>> Hi
>>>>
>>>> Alright, lets give that a shot. My `wait` in global config was set
>>>> as 1800 seconds. I have increased it to 3600 seconds, restarted
>>>> cloudstack-management and started a volume snapshot. Lets see.
>>>> Attached is a screenshot of all wait settings I have in place,
>>>> please let me know if I missed anything?
>>>>
>>>> On 10/18/22 15:31, Nicolas Vazquez wrote:
>>>>> Hi,
>>>>>
>>>>> As a final test I would suggest increasing the value of the ‘wait’ configuration. Checking the logs, it does not ever reach the point in which the snapshot is copied to the snapshots directory on primary storage and it may be taking too long. Currently, the operation is being timeout after 3600 seconds (2 * wait value?), can you try increasing it? It would require restarting the management server after applying the new value. If this doesn’t help please create a new issue on Github
>>>>>
>>>>> Regards,
>>>>> Nicolas Vazquez
>>>>>
>>>>>
>>>>> From: Granwille Strauss<gr...@namhost.com>
>>>>> Date: Tuesday, 18 October 2022 at 09:31
>>>>> To: Wei ZHOU<us...@gmail.com>
>>>>> Cc: Nicolas Vazquez<Ni...@shapeblue.com>,users@cloudstack.apache.org <us...@cloudstack.apache.org>
>>>>> Subject: Re: Recurring Snaps Failing
>>>>>
>>>>> Hi Guys
>>>>>
>>>>> Was there anything else I could do to get this fixed? Should I report this on Github? Also, is it possible to make recurring VM Snapshots and not only Volume Snapshots?
>>>>> On 10/18/22 11:43, Granwille Strauss wrote:
>>>>>
>>>>> Hi Wei
>>>>>
>>>>> Correct, yes VM Snapshot is 100% successful, its only volume snapshot that fails and its at the conversion it seems. As requested:
>>>>> root@HOSTNAME ~ $ virsh domblklist i-2-19-VM
>>>>> Target Source
>>>>> ------------------------------------------------------------------------------------------
>>>>> vda /mnt/81ffca3a-9775-375d-a1c0-9504c0ec3d89/635fd5b5-bf14-4cf4-979c-be8a649d5189
>>>>> hdc -
>>>>>
>>>>> root@HOSTNAME ~ $ qemu-img info /mnt/81ffca3a-9775-375d-a1c0-9504c0ec3d89/635fd5b5-bf14-4cf4-979c-be8a649d5189
>>>>> qemu-img: Could not open '/mnt/81ffca3a-9775-375d-a1c0-9504c0ec3d89/635fd5b5-bf14-4cf4-979c-be8a649d5189': Failed to get shared "write" lock
>>>>> Is another process using the image [/mnt/81ffca3a-9775-375d-a1c0-9504c0ec3d89/635fd5b5-bf14-4cf4-979c-be8a649d5189]?
>>>>> root@athena ~ $
>>>>>
>>>>> Seems I cannot get you the details, however would this help?
>>>>> root@HOSTNAME /mnt/81ffca3a-9775-375d-a1c0-9504c0ec3d89 $ du * -sh
>>>>> 48G 340cc676-4683-4424-beb7-8b33a5cb3dfe
>>>>> 42G 4a56546f-4f63-41c2-a314-27619fe5c2dc
>>>>> 15G 635fd5b5-bf14-4cf4-979c-be8a649d5189
>>>>> 21G 70ffef79-3aad-43ab-93d1-153fcd34ad15
>>>>> 26G 777b5149-50c0-4c0d-944b-873050c85419
>>>>> 20G 867a61e8-4635-40e4-b1b1-c94b2a325b49
>>>>> 4.8G b21551c6-8276-4e58-841e-4e5a02c0e163
>>>>> 11G d2c4feca-da5c-44b3-8ecc-40a01fd6c243
>>>>> 40G f50c0bca-3ae2-48ee-aa9d-0e2d4ec6ba92
>>>>> 24G f9336d09-0b3d-42d8-9a24-883b022b6712
>>>>> 8.9G f962cb2c-e03b-4ed2-a035-1870a367aa46
>>>>> 47G fd573618-885e-48c7-86e6-b019e2311d76
>>>>> 8.0K KVMHA
>>>>> 15G snapshots
>>>>> root@HOSTNAME /mnt/81ffca3a-9775-375d-a1c0-9504c0ec3d89 $
>>>>>
>>>>>
>>>>> I have also added latest agent.log from today.
>>>>> On 10/18/22 11:07, Wei ZHOU wrote:
>>>>> Hi,
>>>>>
>>>>> You can get the path of qcow2 image by "virsh domblklist i-2-19-VM". The QCOW2 image does not have .qcow2 as a suffix.
>>>>>
>>>>> From what I see, a vm snapshot has been created 2022-10-18 09:44:21 +0200, which is good as taking the snapshot started at the same time.
>>>>> so timed out when converting the vm snapshot to a qcow2 image (ons secondary storage) by "qemu-img convert".
>>>>>
>>>>> ps: the agent.log you attached is out-of-date. Please share the agent.log in few minutes after "2022-10-18 09:44:21"
>>>>>
>>>>>
>>>>> -Wei
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Tue, 18 Oct 2022 at 10:53, Granwille Strauss <gr...@namhost.com>> wrote:
>>>>>
>>>>> Hi Wei
>>>>>
>>>>> Thank you for the update. Thus far all the snapshot tests I took were volume snaps and majority were when they are online and end up failing. However, there's one VM where it seems to work just fine, whether its online of offline and I am not sure what's different from this VM as all other VM use the same Almalinux template. Its just this one VM where it works as expected.
>>>>>
>>>>> In any case, I did a VM snapshot now with memory on my Ubuntu vm while it was online and the VM Snap was successful, no issues there. I ran a volume snap right after on the same ubuntu vm and it failed again as before, see attached screenshot. So VM snapshots seem to work fine, this tells me the issue is most likely when its converted to QCOW2.
>>>>>
>>>>> In any case, please see answers to your questions below:
>>>>>
>>>>> 1. I am not finding any .qcow2 images on the KVM server to list the QCOW2 size, or am I missing something?
>>>>> root@HOSTNAME ~ $ find / -type f -iname '*.qcow2'
>>>>> root@HOSTNAME ~ $
>>>>> As requested, here's a snapshot list of the current vm I am testing on. Please let me know if you needed something specific?
>>>>> root@HOSTNAME ~ $ virsh snapshot-list i-2-19-VM
>>>>> Name Creation Time State
>>>>> -----------------------------------------------------------------------------------
>>>>> 0843e6b1-cb12-495c-8d5f-35c9a3338db7 2022-10-17 18:41:08 +0200 disk-snapshot
>>>>> 2e77acb7-1be4-4417-858e-a1d0890f44f0 2022-10-17 19:57:03 +0200 disk-snapshot
>>>>> 68476e2a-aec8-4637-9c4b-2dc49d79bd82 2022-10-18 09:44:21 +0200 disk-snapshot
>>>>> c110058b-4661-42c0-b648-a54f4ba4beb7 2022-10-17 22:34:21 +0200 disk-snapshot
>>>>> i-2-19-VM_VS_20221018072924 2022-10-18 09:29:25 +0200 running
>>>>>
>>>>> 2. Attached is a txt file from yesterdays agent.log. This is all entries that happened between CopyCommand and CopyCmdAnswer. There are zero errors, as you will see.
>>>>>
>>>>> 3. The 'backup.snapshot.wait' setting is currently at 21600 seconds = 6 hours, should this still be increased? Also, not sure if 'copy.volume.wait' is relevant, but this is currently at 10800 seconds.
>>>>> On 10/18/22 09:30, Wei ZHOU wrote:
>>>>> Hi Granwille,
>>>>>
>>>>> The process of volume snapshot on running vms and stop vms are different. For running vms, cloudstack will first take a vm snapshot and convert the vm snapshot into a qcow2 image.
>>>>> It is normal that snapshotting a running vm takes longer time than a stopped vm, because
>>>>> (1) there is an extra step to take a vm snapshot. Depending on the memory size, the step might take a few minutes.
>>>>> (3) vm snapshot is saved in QCOW2 image (it is called Internal snapshots). When taking a vm snapshot, the QCOW2 image will enlarge. The image will not shrink when the vm snapshot is removed.
>>>>> If you have taken vm snapshot (or volume snapshot on running vms) multiple times, The QCOW2 image might be very big. The whole process (including vm snapshotting and qemu-img convert) might exceed the timeout limit (3600 seconds = 1 hour).
>>>>>
>>>>> can you please
>>>>> (1) check the QCOW2 image size and list the snapshot of QCOW2 image (qemu-img info), list vm snapshot of the vm (virsh snapshot-list)
>>>>> (2) check the logs between CopyCommand and CopyCmdAnswer in agent.log
>>>>> (3) increase backup.snapshot.wait in global configuration and retry ("wait" in "CopyCommand" should be changed accordingly) ?
>>>>>
>>>>>
>>>>> -Wei
>>>>>
>>>>>
>>>>>
>>>>> On Tue, 18 Oct 2022 at 00:39, Granwille Strauss <gr...@namhost.com>> wrote:
>>>>>
>>>>> Hi Guys
>>>>>
>>>>> I tested another Linux VM Almalinux VM, turned off the VM and made a successful snap. I turned it back online again and run another successful snap again. Now I run a snap on my Ubuntu VM and it failed. So just to be sure it wasn't just that one specific Almalinux VM that was working, I ran another snap on another Almalinux VM while it was online and too also failed. So I am not sure why recurring snaps is only working for this one VM and not the rest.
>>>>>
>>>>> Should I log this on GITHUB?
>>>>> On 10/17/22 21:55, Granwille Strauss wrote:
>>>>>
>>>>> Sorry, seems my attachments were too big, you can download via Wetransfer:https://we.tl/t-9hUekbj86S
>>>>>
>>>>> And yes, I have the kvm.snapshot.enabled enabled in global settings too, for live snapping.
>>>>> On 10/17/22 21:51, Granwille Strauss wrote:
>>>>>
>>>>> So it seems when VM is powered off, snap is successful. See attached logs for the successful snap, when VM is offline.
>>>>> On 10/17/22 21:38, Granwille Strauss wrote:
>>>>>
>>>>> Here's the last 10 minutes before the error is triggered in the logs, this was the test run before switching off VM. That test is still running and will provide its logs shortly, however, I am confident, it will show same results.
>>>>> On 10/17/22 21:22, Granwille Strauss wrote:
>>>>>
>>>>> Hi Wei
>>>>>
>>>>> Please see attached. Please keep in mind, this worked 100% a few days ago when I tested on VM recurring snaps and it was perfect then. So I enabled recurring snaps for all 10 linux VMs I host. And I set them all to daily at 2AM.
>>>>> They failed after the first day. So I figured maybe too many concurrent jobs running at the same time.
>>>>>
>>>>> So I split them all to once weekly at different days at different hours. And still it failed. And now when doing it manually it fails too. But for some reason the snap ends up in /primary location is just returns an error for some reason.
>>>>> On 10/17/22 21:13, Wei ZHOU wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>>
>>>>>
>>>>> What is the kvm host os and version ?
>>>>>
>>>>>
>>>>>
>>>>> -Wei
>>>>>
>>>>>
>>>>>
>>>>> On Monday, 17 October 2022, Granwille Strauss<gr...@namhost.com.invalid>
>>>>>
>>>>> wrote:
>>>>>
>>>>>
>>>>>
>>>>> The error seems to happen exactly at 1 hour intervals. All of them have
>>>>>
>>>>> failed exactly at one hour. Also, these are all volume snapshots that I am
>>>>>
>>>>> taking, so the logs I sent you so far are all in this regard. They are not
>>>>>
>>>>> VM snaphots, nor are they powered off. I am running a new volume snap shot
>>>>>
>>>>> again, this time VM powered off lets see.
>>>>>
>>>>>
>>>>>
>>>>> I have also rebooted my entire KVM server and my management server. So far
>>>>>
>>>>> zero luck.
>>>>>
>>>>> On 10/17/22 20:26, Nicolas Vazquez wrote:
>>>>>
>>>>>
>>>>>
>>>>> Can you try taking a manual volume snapshot and share both mgmt and agent logs please?
>>>>>
>>>>>
>>>>>
>>>>> Regards,
>>>>>
>>>>> Nicolas Vazquez
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> From: Granwille Strauss<gr...@namhost.com> <gr...@namhost.com>
>>>>>
>>>>> Date: Monday, 17 October 2022 at 13:15
>>>>>
>>>>> To:users@cloudstack.apache.org<ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org>
>>>>>
>>>>> Cc: Nicolas Vazquez<Ni...@shapeblue.com> <Ni...@shapeblue.com>
>>>>>
>>>>> Subject: Re: Recurring Snaps Failing
>>>>>
>>>>>
>>>>>
>>>>> Hi
>>>>>
>>>>>
>>>>>
>>>>> I also see another issue with snaps, I deleted all the failed snaps via the cloud management UI. But to my surprise, they're all still at /mnt/primary/snapshots/ see attached screenshot. Something odd is going on with snapshots.
>>>>>
>>>>> On 10/17/22 17:35, Granwille Strauss wrote:
>>>>>
>>>>>
>>>>>
>>>>> Hi
>>>>>
>>>>>
>>>>>
>>>>> I am not 100% if I enabled debug correctly, I followed thishttps://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+agent+debug and ran
>>>>>
>>>>> sed -i 's/INFO/DEBUG/g' /etc/cloudstack/agent/log4j-cloud.xml
>>>>>
>>>>>
>>>>>
>>>>> Attached are the two log files. Same error. I reboot SSVM & vRouter prior no luck.
>>>>>
>>>>> On 10/17/22 16:00, Granwille Strauss wrote:
>>>>>
>>>>>
>>>>>
>>>>> Hi
>>>>>
>>>>>
>>>>>
>>>>> No KVM is on its own server node and management on a different server node. However, management is is primary and secondary storage via NFS. I will enable DEBUG and re-run manual snap to replicate the error.
>>>>>
>>>>> On 10/17/22 15:42, Nicolas Vazquez wrote:
>>>>>
>>>>> Thanks, are the KVM agent and management server running on the same host? Can you increase the log level on the KVM agent logs to DEBUG and retry?
>>>>>
>>>>>
>>>>>
>>>>> Regards,
>>>>>
>>>>> Nicolas Vazquez
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> From: Granwille Strauss<gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com>
>>>>>
>>>>> Date: Monday, 17 October 2022 at 10:04
>>>>>
>>>>> To:users@cloudstack.apache.org<ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org>
>>>>>
>>>>> Cc: Nicolas Vazquez<Ni...@shapeblue.com> <Ni...@shapeblue.com> <Ni...@shapeblue.com>
>>>>>
>>>>> Subject: Re: Recurring Snaps Failing
>>>>>
>>>>>
>>>>>
>>>>> Please find attached TXT
>>>>>
>>>>> On 10/17/22 14:54, Nicolas Vazquez wrote:
>>>>>
>>>>>
>>>>>
>>>>> Ok, seems like the command is not reaching the KVM host, can you share the full management server logs for job-796?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Regards,
>>>>>
>>>>>
>>>>>
>>>>> Nicolas Vazquez
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> From: Granwille Strauss<gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com>
>>>>>
>>>>>
>>>>>
>>>>> Date: Monday, 17 October 2022 at 09:32
>>>>>
>>>>>
>>>>>
>>>>> To:users@cloudstack.apache.org<ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org>
>>>>>
>>>>>
>>>>>
>>>>> Cc: Nicolas Vazquez<Ni...@shapeblue.com> <Ni...@shapeblue.com> <Ni...@shapeblue.com>
>>>>>
>>>>>
>>>>>
>>>>> Subject: Re: Recurring Snaps Failing
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Hi
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> This is all agent threw during the snap process nothing else:
>>>>>
>>>>>
>>>>>
>>>>> 2022-10-17 13:39:12,375 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:a864f9f6) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
>>>>>
>>>>>
>>>>>
>>>>> 2022-10-17 13:40:13,276 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:445216f3) Trying to fetch storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297 from libvirt
>>>>>
>>>>>
>>>>>
>>>>> 2022-10-17 13:40:13,289 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:445216f3) Asking libvirt to refresh storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297
>>>>>
>>>>>
>>>>>
>>>>> 2022-10-17 13:40:13,378 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:445216f3) Trying to fetch storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89 from libvirt
>>>>>
>>>>>
>>>>>
>>>>> 2022-10-17 13:40:13,393 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:445216f3) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
>>>>>
>>>>>
>>>>>
>>>>> 2022-10-17 13:41:13,950 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:87ecfee0) Trying to fetch storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297 from libvirt
>>>>>
>>>>>
>>>>>
>>>>> 2022-10-17 13:41:13,964 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:87ecfee0) Asking libvirt to refresh storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297
>>>>>
>>>>>
>>>>>
>>>>> 2022-10-17 13:41:14,054 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:87ecfee0) Trying to fetch storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89 from libvirt
>>>>>
>>>>>
>>>>>
>>>>> 2022-10-17 13:41:14,067 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:87ecfee0) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
>>>>>
>>>>>
>>>>>
>>>>> 2022-10-17 13:41:57,753 WARN [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-3:null) (logid:d91c35e5) Expected 1 answers while executing SetMonitorServiceCommand but received 3
>>>>>
>>>>>
>>>>>
>>>>> 2022-10-17 13:41:57,938 INFO [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:null) (logid:1549c9a1) Fetching health check result for SEVERIPADDRESS and executing fresh checks: false
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 10/17/22 14:29, Nicolas Vazquez wrote:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Hi Groete,
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Can you share the CloudStack agent logs from the KVM host?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Regards,
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Nicolas Vazquez
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> From: Granwille Strauss<gr...@namhost.com.INVALID> <gr...@namhost.com.INVALID> <gr...@namhost.com.INVALID> <gr...@namhost.com.INVALID> <gr...@namhost.com.INVALID>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Date: Monday, 17 October 2022 at 08:56
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> To:users@cloudstack.apache.org<ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Subject: Re: Recurring Snaps Failing
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Hi
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> I ran a few more tests and can now finally make sense of the failure:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> 2022-10-17 13:39:08,556 DEBUG [o.a.c.s.d.d.CloudStackPrimaryDataStoreDriverImpl] (Work-Job-Executor-1:ctx-93cb161b job-806/job-807 ctx-3f9ccf45) (logid:8fd10b20) Failed to take snapshot: 23
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> 2022-10-17 13:39:08,558 DEBUG [o.a.c.s.s.SnapshotServiceImpl] (Work-Job-Executor-1:ctx-93cb161b job-806/job-807 ctx-3f9ccf45) (logid:8fd10b20) create snapshot Test001 failed: com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:1, com.cloud.exception.OperationTimedoutException: Commands 7160441932542378205 to Host 1 timed out after 3600
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Seems to be a time out error. But there's no firewalls on the KVM host and NFS server to prevent connections and it worked fine before and randomly just stopped.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> So I still have no idea where to solve this as all other necessary settings are in place. Any ideas?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 10/17/22 08:51, Granwille Strauss wrote:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Hi Guys
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> I enabled recurring snap shots for my Linux VMs and they all seem to be failing with the following error:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> root@hostname ~ $ grep error /var/log/cloudstack/management/management-server.log
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> 2022-10-17 03:01:29,474 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-27:ctx-99d7d702 job-796 ctx-f26a81fe) (logid:f9a61b00) Failed to create snapshot due to an internal error creating snapshot for volume 635fd5b5-bf14-4cf4-979c-be8a649d5189
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> 2022-10-17 03:01:29,476 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-27:ctx-99d7d702 job-796) (logid:f9a61b00) Complete async job-796, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 635fd5b5-bf14-4cf4-979c-be8a649d5189"}
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> 2022-10-17 03:01:29,478 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-26:ctx-c7865967 job-794 ctx-0700afb0) (logid:5541113c) Failed to create snapshot due to an internal error creating snapshot for volume 70ffef79-3aad-43ab-93d1-153fcd34ad15
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> 2022-10-17 03:01:29,480 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-26:ctx-c7865967 job-794) (logid:5541113c) Complete async job-794, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 70ffef79-3aad-43ab-93d1-153fcd34ad15"}
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> All I am getting out of this is internal error and nothing else. I have checked other logs and I am not quite sure where else to look.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Any assistance in this regard will be appreciated.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Regards / Groete
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Granwille Strauss // Senior Systems Admin
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> e:granwille@namhost.com<ma...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> m: +264 81 323 1260<tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> w:www.namhost.com<http://www.namhost.com><http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com><http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com><http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Namhost Internet Services (Pty) Ltd,
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g><https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Regards / Groete
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Granwille Strauss // Senior Systems Admin
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> e:granwille@namhost.com<ma...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> m: +264 81 323 1260<tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> w:www.namhost.com<http://www.namhost.com><http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com><http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com><http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Namhost Internet Services (Pty) Ltd,
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g><https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Regards / Groete
>>>>>
>>>>>
>>>>>
>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
>>>>>
>>>>>
>>>>>
>>>>> Granwille Strauss // Senior Systems Admin
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> e:granwille@namhost.com<ma...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com>
>>>>>
>>>>>
>>>>>
>>>>> m: +264 81 323 1260<tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260>
>>>>>
>>>>>
>>>>>
>>>>> w:www.namhost.com<http://www.namhost.com><http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>>
>>>>>
>>>>>
>>>>> Namhost Internet Services (Pty) Ltd,
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g><https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Regards / Groete
>>>>>
>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
>>>>>
>>>>> Granwille Strauss // Senior Systems Admin
>>>>>
>>>>>
>>>>>
>>>>> e:granwille@namhost.com<ma...@namhost.com> <gr...@namhost.com>
>>>>>
>>>>> m: +264 81 323 1260<tel:+264813231260><tel:+264813231260>
>>>>>
>>>>> w:www.namhost.com<http://www.namhost.com><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
>>>>>
>>>>>
>>>>>
>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>>
>>>>>
>>>>>
>>>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>>
>>>>> Namhost Internet Services (Pty) Ltd,
>>>>>
>>>>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g><https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>>>>
>>>>>
>>>>>
>>>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>>>
>>>>>
>>>>>
>>>>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>>>
>>>>> --
>>>>>
>>>>>
>>>>>
>>>>> Regards / Groete
>>>>>
>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
>>>>>
>>>>> Granwille Strauss // Senior Systems Admin
>>>>>
>>>>>
>>>>>
>>>>> e:granwille@namhost.com<ma...@namhost.com> <gr...@namhost.com>
>>>>>
>>>>> m: +264 81 323 1260<tel:+264813231260><tel:+264813231260>
>>>>>
>>>>> w:www.namhost.com<http://www.namhost.com><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
>>>>>
>>>>>
>>>>>
>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>>
>>>>>
>>>>>
>>>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>>
>>>>> Namhost Internet Services (Pty) Ltd,
>>>>>
>>>>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g><https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>>>>
>>>>>
>>>>>
>>>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and s
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Regards / Groete
>>>>>
>>>>>
>>>>>
>>>>> <https://www.namhost.com><https://www.namhost.com> Granwille Strauss // Senior Systems Admin
>>>>>
>>>>>
>>>>>
>>>>> *e:*granwille@namhost.com<ma...@namhost.com>
>>>>>
>>>>> *m:* +264 81 323 1260 <+264813231260>
>>>>>
>>>>> *w:*www.namhost.com<http://www.namhost.com>
>>>>>
>>>>>
>>>>>
>>>>> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost>
>>>>>
>>>>> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/>
>>>>>
>>>>> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos>
>>>>>
>>>>> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>>
>>>>>
>>>>>
>>>>> Namhost Internet Services (Pty) Ltd,
>>>>>
>>>>>
>>>>>
>>>>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g><https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> The content of this message is confidential. If you have received it by
>>>>>
>>>>> mistake, please inform us by email reply and then delete the message. It is
>>>>>
>>>>> forbidden to copy, forward, or in any way reveal the contents of this
>>>>>
>>>>> message to anyone without our explicit consent. The integrity and security
>>>>>
>>>>> of this email cannot be guaranteed over the Internet. Therefore, the sender
>>>>>
>>>>> will not be held liable for any damage caused by the message. For our full
>>>>>
>>>>> privacy policy and disclaimers, please go tohttps://www.namhost.com/
>>>>>
>>>>> privacy-policy
>>>>>
>>>>>
>>>>>
>>>>> [image: Powered by AdSigner]
>>>>>
>>>>> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>>>
>>>>>
>>>>> --
>>>>> Regards / Groete
>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
>>>>> Granwille Strauss // Senior Systems Admin
>>>>>
>>>>> e:granwille@namhost.com<ma...@namhost.com>
>>>>> m: +264 81 323 1260<tel:+264813231260>
>>>>> w:www.namhost.com<https://www.namhost.com/>
>>>>>
>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>>
>>>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>> Namhost Internet Services (Pty) Ltd,
>>>>>
>>>>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>>>>
>>>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>>>
>>>>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>>> --
>>>>> Regards / Groete
>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
>>>>> Granwille Strauss // Senior Systems Admin
>>>>>
>>>>> e:granwille@namhost.com<ma...@namhost.com>
>>>>> m: +264 81 323 1260<tel:+264813231260>
>>>>> w:www.namhost.com<https://www.namhost.com/>
>>>>>
>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>>
>>>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>> Namhost Internet Services (Pty) Ltd,
>>>>>
>>>>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>>>>
>>>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>>>
>>>>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>>> --
>>>>> Regards / Groete
>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
>>>>> Granwille Strauss // Senior Systems Admin
>>>>>
>>>>> e:granwille@namhost.com<ma...@namhost.com>
>>>>> m: +264 81 323 1260<tel:+264813231260>
>>>>> w:www.namhost.com<https://www.namhost.com/>
>>>>>
>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>>
>>>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>> Namhost Internet Services (Pty) Ltd,
>>>>>
>>>>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>>>>
>>>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>>>
>>>>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>>> --
>>>>> Regards / Groete
>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
>>>>> Granwille Strauss // Senior Systems Admin
>>>>>
>>>>> e:granwille@namhost.com<ma...@namhost.com>
>>>>> m: +264 81 323 1260<tel:+264813231260>
>>>>> w:www.namhost.com<https://www.namhost.com/>
>>>>>
>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>>
>>>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>> Namhost Internet Services (Pty) Ltd,
>>>>>
>>>>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>>>>
>>>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>>>
>>>>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>>> --
>>>>> Regards / Groete
>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
>>>>> Granwille Strauss // Senior Systems Admin
>>>>>
>>>>> e:granwille@namhost.com<ma...@namhost.com>
>>>>> m: +264 81 323 1260<tel:+264813231260>
>>>>> w:www.namhost.com<https://www.namhost.com/>
>>>>>
>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>>
>>>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>> Namhost Internet Services (Pty) Ltd,
>>>>>
>>>>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>>>>
>>>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>>>
>>>>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>>> --
>>>>> Regards / Groete
>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
>>>>> Granwille Strauss // Senior Systems Admin
>>>>>
>>>>> e:granwille@namhost.com<ma...@namhost.com>
>>>>> m: +264 81 323 1260<tel:+264813231260>
>>>>> w:www.namhost.com<https://www.namhost.com/>
>>>>>
>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>>
>>>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>> Namhost Internet Services (Pty) Ltd,
>>>>>
>>>>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>>>>
>>>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>>>
>>>>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>>> --
>>>>>
>>>>> Regards / Groete
>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
>>>>> Granwille Strauss // Senior Systems Admin
>>>>>
>>>>> e:granwille@namhost.com<ma...@namhost.com>
>>>>> m: +264 81 323 1260<tel:+264813231260>
>>>>> w:www.namhost.com<https://www.namhost.com/>
>>>>>
>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>>
>>>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>> Namhost Internet Services (Pty) Ltd,
>>>>>
>>>>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>>>>
>>>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>>>
>>>>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>>> --
>>>>>
>>>>> Regards / Groete
>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
>>>>> Granwille Strauss // Senior Systems Admin
>>>>>
>>>>> e:granwille@namhost.com<ma...@namhost.com>
>>>>> m: +264 81 323 1260<tel:+264813231260>
>>>>> w:www.namhost.com<https://www.namhost.com/>
>>>>>
>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>>
>>>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>> Namhost Internet Services (Pty) Ltd,
>>>>>
>>>>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>>>>
>>>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>>>
>>>>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>>>
>>>> --
>>>> Regards / Groete
>>>>
>>>> <https://www.namhost.com> Granwille Strauss // Senior Systems Admin
>>>>
>>>> *e:* granwille@namhost.com
>>>> *m:* +264 81 323 1260 <tel:+264813231260>
>>>> *w:* www.namhost.com <https://www.namhost.com/>
>>>>
>>>> <https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>
>>>> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>
>>>> Namhost Internet Services (Pty) Ltd,
>>>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>>>
>>>>
>>>> The content of this message is confidential. If you have received
>>>> it by mistake, please inform us by email reply and then delete the
>>>> message. It is forbidden to copy, forward, or in any way reveal the
>>>> contents of this message to anyone without our explicit consent.
>>>> The integrity and security of this email cannot be guaranteed over
>>>> the Internet. Therefore, the sender will not be held liable for any
>>>> damage caused by the message. For our full privacy policy and
>>>> disclaimers, please go to https://www.namhost.com/privacy-policy
>>>>
>>>> Powered by AdSigner
>>>> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>> --
>>> Regards / Groete
>>>
>>> <https://www.namhost.com> Granwille Strauss // Senior Systems Admin
>>>
>>> *e:* granwille@namhost.com
>>> *m:* +264 81 323 1260 <tel:+264813231260>
>>> *w:* www.namhost.com <https://www.namhost.com/>
>>>
>>> <https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>
>>> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>
>>> Namhost Internet Services (Pty) Ltd,
>>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>>
>>>
>>> The content of this message is confidential. If you have received it
>>> by mistake, please inform us by email reply and then delete the
>>> message. It is forbidden to copy, forward, or in any way reveal the
>>> contents of this message to anyone without our explicit consent. The
>>> integrity and security of this email cannot be guaranteed over the
>>> Internet. Therefore, the sender will not be held liable for any
>>> damage caused by the message. For our full privacy policy and
>>> disclaimers, please go to https://www.namhost.com/privacy-policy
>>>
>>> Powered by AdSigner
>>> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>> --
>> Regards / Groete
>>
>> <https://www.namhost.com> Granwille Strauss // Senior Systems Admin
>>
>> *e:* granwille@namhost.com
>> *m:* +264 81 323 1260 <tel:+264813231260>
>> *w:* www.namhost.com <https://www.namhost.com/>
>>
>> <https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>
>> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>
>> Namhost Internet Services (Pty) Ltd,
>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>
>>
>> The content of this message is confidential. If you have received it
>> by mistake, please inform us by email reply and then delete the
>> message. It is forbidden to copy, forward, or in any way reveal the
>> contents of this message to anyone without our explicit consent. The
>> integrity and security of this email cannot be guaranteed over the
>> Internet. Therefore, the sender will not be held liable for any
>> damage caused by the message. For our full privacy policy and
>> disclaimers, please go to https://www.namhost.com/privacy-policy
>>
>> Powered by AdSigner
>> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
> --
> Regards / Groete
>
> <https://www.namhost.com> Granwille Strauss // Senior Systems Admin
>
> *e:* granwille@namhost.com
> *m:* +264 81 323 1260 <tel:+264813231260>
> *w:* www.namhost.com <https://www.namhost.com/>
>
> <https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>
> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>
> Namhost Internet Services (Pty) Ltd,
> 24 Black Eagle Rd, Hermanus, 7210, RSA
>
>
> The content of this message is confidential. If you have received it
> by mistake, please inform us by email reply and then delete the
> message. It is forbidden to copy, forward, or in any way reveal the
> contents of this message to anyone without our explicit consent. The
> integrity and security of this email cannot be guaranteed over the
> Internet. Therefore, the sender will not be held liable for any damage
> caused by the message. For our full privacy policy and disclaimers,
> please go to https://www.namhost.com/privacy-policy
>
> Powered by AdSigner
> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
--
Regards / Groete
<https://www.namhost.com> Granwille Strauss // Senior Systems Admin
*e:* granwille@namhost.com
*m:* +264 81 323 1260 <tel:+264813231260>
*w:* www.namhost.com <https://www.namhost.com/>
<https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
Namhost Internet Services (Pty) Ltd,
24 Black Eagle Rd, Hermanus, 7210, RSA
The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It
is forbidden to copy, forward, or in any way reveal the contents of this
message to anyone without our explicit consent. The integrity and
security of this email cannot be guaranteed over the Internet.
Therefore, the sender will not be held liable for any damage caused by
the message. For our full privacy policy and disclaimers, please go to
https://www.namhost.com/privacy-policy
Powered by AdSigner
<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
Re: Recurring Snaps Failing
Posted by Granwille Strauss <gr...@namhost.com.INVALID>.
For anyone following this thread, because my KVM server and Cloud
Management server is connected via public network, the speeds were too
slow to complete the volume snapshots in time and caused a time out at
3600 seconds. As a solution, we increased `job.cancel.threshold.minutes`
in the global settings from the default 60 minutes to 120 minute, which
seems to have solved the issue.
However, the proper solution is to implement a good LAN network between
the two servers instead.
On 10/18/22 17:51, Granwille Strauss wrote:
>
> Hi Guys
>
> I have logged the issue on GITHub:
> https://github.com/apache/cloudstack/issues/6836 I would appreciate it
> if you can take some time to comment on the issue on GITHub, to help
> better explain what the problem is, please. Thank you very much for
> your assistance and patience thus far. Hopefully we can get a solution
> soon.
>
> On 10/18/22 17:04, Granwille Strauss wrote:
>>
>> Hi
>>
>> It failed again right after 1 hour, even after I changed the 'wait'
>> from 1800 to 3600. Is there not another "wait" setting that may need
>> to be increased?
>>
>> On 10/18/22 15:38, Granwille Strauss wrote:
>>>
>>> Hi
>>>
>>> Alright, lets give that a shot. My `wait` in global config was set
>>> as 1800 seconds. I have increased it to 3600 seconds, restarted
>>> cloudstack-management and started a volume snapshot. Lets see.
>>> Attached is a screenshot of all wait settings I have in place,
>>> please let me know if I missed anything?
>>>
>>> On 10/18/22 15:31, Nicolas Vazquez wrote:
>>>> Hi,
>>>>
>>>> As a final test I would suggest increasing the value of the ‘wait’ configuration. Checking the logs, it does not ever reach the point in which the snapshot is copied to the snapshots directory on primary storage and it may be taking too long. Currently, the operation is being timeout after 3600 seconds (2 * wait value?), can you try increasing it? It would require restarting the management server after applying the new value. If this doesn’t help please create a new issue on Github
>>>>
>>>> Regards,
>>>> Nicolas Vazquez
>>>>
>>>>
>>>> From: Granwille Strauss<gr...@namhost.com>
>>>> Date: Tuesday, 18 October 2022 at 09:31
>>>> To: Wei ZHOU<us...@gmail.com>
>>>> Cc: Nicolas Vazquez<Ni...@shapeblue.com>,users@cloudstack.apache.org <us...@cloudstack.apache.org>
>>>> Subject: Re: Recurring Snaps Failing
>>>>
>>>> Hi Guys
>>>>
>>>> Was there anything else I could do to get this fixed? Should I report this on Github? Also, is it possible to make recurring VM Snapshots and not only Volume Snapshots?
>>>> On 10/18/22 11:43, Granwille Strauss wrote:
>>>>
>>>> Hi Wei
>>>>
>>>> Correct, yes VM Snapshot is 100% successful, its only volume snapshot that fails and its at the conversion it seems. As requested:
>>>> root@HOSTNAME ~ $ virsh domblklist i-2-19-VM
>>>> Target Source
>>>> ------------------------------------------------------------------------------------------
>>>> vda /mnt/81ffca3a-9775-375d-a1c0-9504c0ec3d89/635fd5b5-bf14-4cf4-979c-be8a649d5189
>>>> hdc -
>>>>
>>>> root@HOSTNAME ~ $ qemu-img info /mnt/81ffca3a-9775-375d-a1c0-9504c0ec3d89/635fd5b5-bf14-4cf4-979c-be8a649d5189
>>>> qemu-img: Could not open '/mnt/81ffca3a-9775-375d-a1c0-9504c0ec3d89/635fd5b5-bf14-4cf4-979c-be8a649d5189': Failed to get shared "write" lock
>>>> Is another process using the image [/mnt/81ffca3a-9775-375d-a1c0-9504c0ec3d89/635fd5b5-bf14-4cf4-979c-be8a649d5189]?
>>>> root@athena ~ $
>>>>
>>>> Seems I cannot get you the details, however would this help?
>>>> root@HOSTNAME /mnt/81ffca3a-9775-375d-a1c0-9504c0ec3d89 $ du * -sh
>>>> 48G 340cc676-4683-4424-beb7-8b33a5cb3dfe
>>>> 42G 4a56546f-4f63-41c2-a314-27619fe5c2dc
>>>> 15G 635fd5b5-bf14-4cf4-979c-be8a649d5189
>>>> 21G 70ffef79-3aad-43ab-93d1-153fcd34ad15
>>>> 26G 777b5149-50c0-4c0d-944b-873050c85419
>>>> 20G 867a61e8-4635-40e4-b1b1-c94b2a325b49
>>>> 4.8G b21551c6-8276-4e58-841e-4e5a02c0e163
>>>> 11G d2c4feca-da5c-44b3-8ecc-40a01fd6c243
>>>> 40G f50c0bca-3ae2-48ee-aa9d-0e2d4ec6ba92
>>>> 24G f9336d09-0b3d-42d8-9a24-883b022b6712
>>>> 8.9G f962cb2c-e03b-4ed2-a035-1870a367aa46
>>>> 47G fd573618-885e-48c7-86e6-b019e2311d76
>>>> 8.0K KVMHA
>>>> 15G snapshots
>>>> root@HOSTNAME /mnt/81ffca3a-9775-375d-a1c0-9504c0ec3d89 $
>>>>
>>>>
>>>> I have also added latest agent.log from today.
>>>> On 10/18/22 11:07, Wei ZHOU wrote:
>>>> Hi,
>>>>
>>>> You can get the path of qcow2 image by "virsh domblklist i-2-19-VM". The QCOW2 image does not have .qcow2 as a suffix.
>>>>
>>>> From what I see, a vm snapshot has been created 2022-10-18 09:44:21 +0200, which is good as taking the snapshot started at the same time.
>>>> so timed out when converting the vm snapshot to a qcow2 image (ons secondary storage) by "qemu-img convert".
>>>>
>>>> ps: the agent.log you attached is out-of-date. Please share the agent.log in few minutes after "2022-10-18 09:44:21"
>>>>
>>>>
>>>> -Wei
>>>>
>>>>
>>>>
>>>>
>>>> On Tue, 18 Oct 2022 at 10:53, Granwille Strauss <gr...@namhost.com>> wrote:
>>>>
>>>> Hi Wei
>>>>
>>>> Thank you for the update. Thus far all the snapshot tests I took were volume snaps and majority were when they are online and end up failing. However, there's one VM where it seems to work just fine, whether its online of offline and I am not sure what's different from this VM as all other VM use the same Almalinux template. Its just this one VM where it works as expected.
>>>>
>>>> In any case, I did a VM snapshot now with memory on my Ubuntu vm while it was online and the VM Snap was successful, no issues there. I ran a volume snap right after on the same ubuntu vm and it failed again as before, see attached screenshot. So VM snapshots seem to work fine, this tells me the issue is most likely when its converted to QCOW2.
>>>>
>>>> In any case, please see answers to your questions below:
>>>>
>>>> 1. I am not finding any .qcow2 images on the KVM server to list the QCOW2 size, or am I missing something?
>>>> root@HOSTNAME ~ $ find / -type f -iname '*.qcow2'
>>>> root@HOSTNAME ~ $
>>>> As requested, here's a snapshot list of the current vm I am testing on. Please let me know if you needed something specific?
>>>> root@HOSTNAME ~ $ virsh snapshot-list i-2-19-VM
>>>> Name Creation Time State
>>>> -----------------------------------------------------------------------------------
>>>> 0843e6b1-cb12-495c-8d5f-35c9a3338db7 2022-10-17 18:41:08 +0200 disk-snapshot
>>>> 2e77acb7-1be4-4417-858e-a1d0890f44f0 2022-10-17 19:57:03 +0200 disk-snapshot
>>>> 68476e2a-aec8-4637-9c4b-2dc49d79bd82 2022-10-18 09:44:21 +0200 disk-snapshot
>>>> c110058b-4661-42c0-b648-a54f4ba4beb7 2022-10-17 22:34:21 +0200 disk-snapshot
>>>> i-2-19-VM_VS_20221018072924 2022-10-18 09:29:25 +0200 running
>>>>
>>>> 2. Attached is a txt file from yesterdays agent.log. This is all entries that happened between CopyCommand and CopyCmdAnswer. There are zero errors, as you will see.
>>>>
>>>> 3. The 'backup.snapshot.wait' setting is currently at 21600 seconds = 6 hours, should this still be increased? Also, not sure if 'copy.volume.wait' is relevant, but this is currently at 10800 seconds.
>>>> On 10/18/22 09:30, Wei ZHOU wrote:
>>>> Hi Granwille,
>>>>
>>>> The process of volume snapshot on running vms and stop vms are different. For running vms, cloudstack will first take a vm snapshot and convert the vm snapshot into a qcow2 image.
>>>> It is normal that snapshotting a running vm takes longer time than a stopped vm, because
>>>> (1) there is an extra step to take a vm snapshot. Depending on the memory size, the step might take a few minutes.
>>>> (3) vm snapshot is saved in QCOW2 image (it is called Internal snapshots). When taking a vm snapshot, the QCOW2 image will enlarge. The image will not shrink when the vm snapshot is removed.
>>>> If you have taken vm snapshot (or volume snapshot on running vms) multiple times, The QCOW2 image might be very big. The whole process (including vm snapshotting and qemu-img convert) might exceed the timeout limit (3600 seconds = 1 hour).
>>>>
>>>> can you please
>>>> (1) check the QCOW2 image size and list the snapshot of QCOW2 image (qemu-img info), list vm snapshot of the vm (virsh snapshot-list)
>>>> (2) check the logs between CopyCommand and CopyCmdAnswer in agent.log
>>>> (3) increase backup.snapshot.wait in global configuration and retry ("wait" in "CopyCommand" should be changed accordingly) ?
>>>>
>>>>
>>>> -Wei
>>>>
>>>>
>>>>
>>>> On Tue, 18 Oct 2022 at 00:39, Granwille Strauss <gr...@namhost.com>> wrote:
>>>>
>>>> Hi Guys
>>>>
>>>> I tested another Linux VM Almalinux VM, turned off the VM and made a successful snap. I turned it back online again and run another successful snap again. Now I run a snap on my Ubuntu VM and it failed. So just to be sure it wasn't just that one specific Almalinux VM that was working, I ran another snap on another Almalinux VM while it was online and too also failed. So I am not sure why recurring snaps is only working for this one VM and not the rest.
>>>>
>>>> Should I log this on GITHUB?
>>>> On 10/17/22 21:55, Granwille Strauss wrote:
>>>>
>>>> Sorry, seems my attachments were too big, you can download via Wetransfer:https://we.tl/t-9hUekbj86S
>>>>
>>>> And yes, I have the kvm.snapshot.enabled enabled in global settings too, for live snapping.
>>>> On 10/17/22 21:51, Granwille Strauss wrote:
>>>>
>>>> So it seems when VM is powered off, snap is successful. See attached logs for the successful snap, when VM is offline.
>>>> On 10/17/22 21:38, Granwille Strauss wrote:
>>>>
>>>> Here's the last 10 minutes before the error is triggered in the logs, this was the test run before switching off VM. That test is still running and will provide its logs shortly, however, I am confident, it will show same results.
>>>> On 10/17/22 21:22, Granwille Strauss wrote:
>>>>
>>>> Hi Wei
>>>>
>>>> Please see attached. Please keep in mind, this worked 100% a few days ago when I tested on VM recurring snaps and it was perfect then. So I enabled recurring snaps for all 10 linux VMs I host. And I set them all to daily at 2AM.
>>>> They failed after the first day. So I figured maybe too many concurrent jobs running at the same time.
>>>>
>>>> So I split them all to once weekly at different days at different hours. And still it failed. And now when doing it manually it fails too. But for some reason the snap ends up in /primary location is just returns an error for some reason.
>>>> On 10/17/22 21:13, Wei ZHOU wrote:
>>>>
>>>> Hi,
>>>>
>>>>
>>>>
>>>> What is the kvm host os and version ?
>>>>
>>>>
>>>>
>>>> -Wei
>>>>
>>>>
>>>>
>>>> On Monday, 17 October 2022, Granwille Strauss<gr...@namhost.com.invalid>
>>>>
>>>> wrote:
>>>>
>>>>
>>>>
>>>> The error seems to happen exactly at 1 hour intervals. All of them have
>>>>
>>>> failed exactly at one hour. Also, these are all volume snapshots that I am
>>>>
>>>> taking, so the logs I sent you so far are all in this regard. They are not
>>>>
>>>> VM snaphots, nor are they powered off. I am running a new volume snap shot
>>>>
>>>> again, this time VM powered off lets see.
>>>>
>>>>
>>>>
>>>> I have also rebooted my entire KVM server and my management server. So far
>>>>
>>>> zero luck.
>>>>
>>>> On 10/17/22 20:26, Nicolas Vazquez wrote:
>>>>
>>>>
>>>>
>>>> Can you try taking a manual volume snapshot and share both mgmt and agent logs please?
>>>>
>>>>
>>>>
>>>> Regards,
>>>>
>>>> Nicolas Vazquez
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> From: Granwille Strauss<gr...@namhost.com> <gr...@namhost.com>
>>>>
>>>> Date: Monday, 17 October 2022 at 13:15
>>>>
>>>> To:users@cloudstack.apache.org<ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org>
>>>>
>>>> Cc: Nicolas Vazquez<Ni...@shapeblue.com> <Ni...@shapeblue.com>
>>>>
>>>> Subject: Re: Recurring Snaps Failing
>>>>
>>>>
>>>>
>>>> Hi
>>>>
>>>>
>>>>
>>>> I also see another issue with snaps, I deleted all the failed snaps via the cloud management UI. But to my surprise, they're all still at /mnt/primary/snapshots/ see attached screenshot. Something odd is going on with snapshots.
>>>>
>>>> On 10/17/22 17:35, Granwille Strauss wrote:
>>>>
>>>>
>>>>
>>>> Hi
>>>>
>>>>
>>>>
>>>> I am not 100% if I enabled debug correctly, I followed thishttps://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+agent+debug and ran
>>>>
>>>> sed -i 's/INFO/DEBUG/g' /etc/cloudstack/agent/log4j-cloud.xml
>>>>
>>>>
>>>>
>>>> Attached are the two log files. Same error. I reboot SSVM & vRouter prior no luck.
>>>>
>>>> On 10/17/22 16:00, Granwille Strauss wrote:
>>>>
>>>>
>>>>
>>>> Hi
>>>>
>>>>
>>>>
>>>> No KVM is on its own server node and management on a different server node. However, management is is primary and secondary storage via NFS. I will enable DEBUG and re-run manual snap to replicate the error.
>>>>
>>>> On 10/17/22 15:42, Nicolas Vazquez wrote:
>>>>
>>>> Thanks, are the KVM agent and management server running on the same host? Can you increase the log level on the KVM agent logs to DEBUG and retry?
>>>>
>>>>
>>>>
>>>> Regards,
>>>>
>>>> Nicolas Vazquez
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> From: Granwille Strauss<gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com>
>>>>
>>>> Date: Monday, 17 October 2022 at 10:04
>>>>
>>>> To:users@cloudstack.apache.org<ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org>
>>>>
>>>> Cc: Nicolas Vazquez<Ni...@shapeblue.com> <Ni...@shapeblue.com> <Ni...@shapeblue.com>
>>>>
>>>> Subject: Re: Recurring Snaps Failing
>>>>
>>>>
>>>>
>>>> Please find attached TXT
>>>>
>>>> On 10/17/22 14:54, Nicolas Vazquez wrote:
>>>>
>>>>
>>>>
>>>> Ok, seems like the command is not reaching the KVM host, can you share the full management server logs for job-796?
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Regards,
>>>>
>>>>
>>>>
>>>> Nicolas Vazquez
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> From: Granwille Strauss<gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com>
>>>>
>>>>
>>>>
>>>> Date: Monday, 17 October 2022 at 09:32
>>>>
>>>>
>>>>
>>>> To:users@cloudstack.apache.org<ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org>
>>>>
>>>>
>>>>
>>>> Cc: Nicolas Vazquez<Ni...@shapeblue.com> <Ni...@shapeblue.com> <Ni...@shapeblue.com>
>>>>
>>>>
>>>>
>>>> Subject: Re: Recurring Snaps Failing
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Hi
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> This is all agent threw during the snap process nothing else:
>>>>
>>>>
>>>>
>>>> 2022-10-17 13:39:12,375 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:a864f9f6) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
>>>>
>>>>
>>>>
>>>> 2022-10-17 13:40:13,276 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:445216f3) Trying to fetch storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297 from libvirt
>>>>
>>>>
>>>>
>>>> 2022-10-17 13:40:13,289 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:445216f3) Asking libvirt to refresh storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297
>>>>
>>>>
>>>>
>>>> 2022-10-17 13:40:13,378 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:445216f3) Trying to fetch storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89 from libvirt
>>>>
>>>>
>>>>
>>>> 2022-10-17 13:40:13,393 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:445216f3) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
>>>>
>>>>
>>>>
>>>> 2022-10-17 13:41:13,950 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:87ecfee0) Trying to fetch storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297 from libvirt
>>>>
>>>>
>>>>
>>>> 2022-10-17 13:41:13,964 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:87ecfee0) Asking libvirt to refresh storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297
>>>>
>>>>
>>>>
>>>> 2022-10-17 13:41:14,054 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:87ecfee0) Trying to fetch storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89 from libvirt
>>>>
>>>>
>>>>
>>>> 2022-10-17 13:41:14,067 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:87ecfee0) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
>>>>
>>>>
>>>>
>>>> 2022-10-17 13:41:57,753 WARN [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-3:null) (logid:d91c35e5) Expected 1 answers while executing SetMonitorServiceCommand but received 3
>>>>
>>>>
>>>>
>>>> 2022-10-17 13:41:57,938 INFO [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:null) (logid:1549c9a1) Fetching health check result for SEVERIPADDRESS and executing fresh checks: false
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On 10/17/22 14:29, Nicolas Vazquez wrote:
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Hi Groete,
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Can you share the CloudStack agent logs from the KVM host?
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Regards,
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Nicolas Vazquez
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> From: Granwille Strauss<gr...@namhost.com.INVALID> <gr...@namhost.com.INVALID> <gr...@namhost.com.INVALID> <gr...@namhost.com.INVALID> <gr...@namhost.com.INVALID>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Date: Monday, 17 October 2022 at 08:56
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> To:users@cloudstack.apache.org<ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Subject: Re: Recurring Snaps Failing
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Hi
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> I ran a few more tests and can now finally make sense of the failure:
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> 2022-10-17 13:39:08,556 DEBUG [o.a.c.s.d.d.CloudStackPrimaryDataStoreDriverImpl] (Work-Job-Executor-1:ctx-93cb161b job-806/job-807 ctx-3f9ccf45) (logid:8fd10b20) Failed to take snapshot: 23
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> 2022-10-17 13:39:08,558 DEBUG [o.a.c.s.s.SnapshotServiceImpl] (Work-Job-Executor-1:ctx-93cb161b job-806/job-807 ctx-3f9ccf45) (logid:8fd10b20) create snapshot Test001 failed: com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:1, com.cloud.exception.OperationTimedoutException: Commands 7160441932542378205 to Host 1 timed out after 3600
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Seems to be a time out error. But there's no firewalls on the KVM host and NFS server to prevent connections and it worked fine before and randomly just stopped.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> So I still have no idea where to solve this as all other necessary settings are in place. Any ideas?
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On 10/17/22 08:51, Granwille Strauss wrote:
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Hi Guys
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> I enabled recurring snap shots for my Linux VMs and they all seem to be failing with the following error:
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> root@hostname ~ $ grep error /var/log/cloudstack/management/management-server.log
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> 2022-10-17 03:01:29,474 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-27:ctx-99d7d702 job-796 ctx-f26a81fe) (logid:f9a61b00) Failed to create snapshot due to an internal error creating snapshot for volume 635fd5b5-bf14-4cf4-979c-be8a649d5189
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> 2022-10-17 03:01:29,476 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-27:ctx-99d7d702 job-796) (logid:f9a61b00) Complete async job-796, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 635fd5b5-bf14-4cf4-979c-be8a649d5189"}
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> 2022-10-17 03:01:29,478 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-26:ctx-c7865967 job-794 ctx-0700afb0) (logid:5541113c) Failed to create snapshot due to an internal error creating snapshot for volume 70ffef79-3aad-43ab-93d1-153fcd34ad15
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> 2022-10-17 03:01:29,480 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-26:ctx-c7865967 job-794) (logid:5541113c) Complete async job-794, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 70ffef79-3aad-43ab-93d1-153fcd34ad15"}
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> All I am getting out of this is internal error and nothing else. I have checked other logs and I am not quite sure where else to look.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Any assistance in this regard will be appreciated.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Regards / Groete
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Granwille Strauss // Senior Systems Admin
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> e:granwille@namhost.com<ma...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> m: +264 81 323 1260<tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> w:www.namhost.com<http://www.namhost.com><http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com><http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com><http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Namhost Internet Services (Pty) Ltd,
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g><https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Regards / Groete
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Granwille Strauss // Senior Systems Admin
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> e:granwille@namhost.com<ma...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> m: +264 81 323 1260<tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> w:www.namhost.com<http://www.namhost.com><http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com><http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com><http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Namhost Internet Services (Pty) Ltd,
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g><https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Regards / Groete
>>>>
>>>>
>>>>
>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
>>>>
>>>>
>>>>
>>>> Granwille Strauss // Senior Systems Admin
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> e:granwille@namhost.com<ma...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com>
>>>>
>>>>
>>>>
>>>> m: +264 81 323 1260<tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260>
>>>>
>>>>
>>>>
>>>> w:www.namhost.com<http://www.namhost.com><http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>
>>>>
>>>>
>>>> Namhost Internet Services (Pty) Ltd,
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g><https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Regards / Groete
>>>>
>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
>>>>
>>>> Granwille Strauss // Senior Systems Admin
>>>>
>>>>
>>>>
>>>> e:granwille@namhost.com<ma...@namhost.com> <gr...@namhost.com>
>>>>
>>>> m: +264 81 323 1260<tel:+264813231260><tel:+264813231260>
>>>>
>>>> w:www.namhost.com<http://www.namhost.com><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
>>>>
>>>>
>>>>
>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>
>>>>
>>>>
>>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>
>>>> Namhost Internet Services (Pty) Ltd,
>>>>
>>>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g><https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>>>
>>>>
>>>>
>>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>>
>>>>
>>>>
>>>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>>
>>>> --
>>>>
>>>>
>>>>
>>>> Regards / Groete
>>>>
>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
>>>>
>>>> Granwille Strauss // Senior Systems Admin
>>>>
>>>>
>>>>
>>>> e:granwille@namhost.com<ma...@namhost.com> <gr...@namhost.com>
>>>>
>>>> m: +264 81 323 1260<tel:+264813231260><tel:+264813231260>
>>>>
>>>> w:www.namhost.com<http://www.namhost.com><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
>>>>
>>>>
>>>>
>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>
>>>>
>>>>
>>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>
>>>> Namhost Internet Services (Pty) Ltd,
>>>>
>>>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g><https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>>>
>>>>
>>>>
>>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and s
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Regards / Groete
>>>>
>>>>
>>>>
>>>> <https://www.namhost.com><https://www.namhost.com> Granwille Strauss // Senior Systems Admin
>>>>
>>>>
>>>>
>>>> *e:*granwille@namhost.com<ma...@namhost.com>
>>>>
>>>> *m:* +264 81 323 1260 <+264813231260>
>>>>
>>>> *w:*www.namhost.com<http://www.namhost.com>
>>>>
>>>>
>>>>
>>>> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost>
>>>>
>>>> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/>
>>>>
>>>> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos>
>>>>
>>>> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>
>>>>
>>>>
>>>> Namhost Internet Services (Pty) Ltd,
>>>>
>>>>
>>>>
>>>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g><https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> The content of this message is confidential. If you have received it by
>>>>
>>>> mistake, please inform us by email reply and then delete the message. It is
>>>>
>>>> forbidden to copy, forward, or in any way reveal the contents of this
>>>>
>>>> message to anyone without our explicit consent. The integrity and security
>>>>
>>>> of this email cannot be guaranteed over the Internet. Therefore, the sender
>>>>
>>>> will not be held liable for any damage caused by the message. For our full
>>>>
>>>> privacy policy and disclaimers, please go tohttps://www.namhost.com/
>>>>
>>>> privacy-policy
>>>>
>>>>
>>>>
>>>> [image: Powered by AdSigner]
>>>>
>>>> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>>
>>>>
>>>> --
>>>> Regards / Groete
>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
>>>> Granwille Strauss // Senior Systems Admin
>>>>
>>>> e:granwille@namhost.com<ma...@namhost.com>
>>>> m: +264 81 323 1260<tel:+264813231260>
>>>> w:www.namhost.com<https://www.namhost.com/>
>>>>
>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>
>>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>> Namhost Internet Services (Pty) Ltd,
>>>>
>>>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>>>
>>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>>
>>>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>> --
>>>> Regards / Groete
>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
>>>> Granwille Strauss // Senior Systems Admin
>>>>
>>>> e:granwille@namhost.com<ma...@namhost.com>
>>>> m: +264 81 323 1260<tel:+264813231260>
>>>> w:www.namhost.com<https://www.namhost.com/>
>>>>
>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>
>>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>> Namhost Internet Services (Pty) Ltd,
>>>>
>>>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>>>
>>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>>
>>>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>> --
>>>> Regards / Groete
>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
>>>> Granwille Strauss // Senior Systems Admin
>>>>
>>>> e:granwille@namhost.com<ma...@namhost.com>
>>>> m: +264 81 323 1260<tel:+264813231260>
>>>> w:www.namhost.com<https://www.namhost.com/>
>>>>
>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>
>>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>> Namhost Internet Services (Pty) Ltd,
>>>>
>>>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>>>
>>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>>
>>>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>> --
>>>> Regards / Groete
>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
>>>> Granwille Strauss // Senior Systems Admin
>>>>
>>>> e:granwille@namhost.com<ma...@namhost.com>
>>>> m: +264 81 323 1260<tel:+264813231260>
>>>> w:www.namhost.com<https://www.namhost.com/>
>>>>
>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>
>>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>> Namhost Internet Services (Pty) Ltd,
>>>>
>>>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>>>
>>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>>
>>>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>> --
>>>> Regards / Groete
>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
>>>> Granwille Strauss // Senior Systems Admin
>>>>
>>>> e:granwille@namhost.com<ma...@namhost.com>
>>>> m: +264 81 323 1260<tel:+264813231260>
>>>> w:www.namhost.com<https://www.namhost.com/>
>>>>
>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>
>>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>> Namhost Internet Services (Pty) Ltd,
>>>>
>>>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>>>
>>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>>
>>>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>> --
>>>> Regards / Groete
>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
>>>> Granwille Strauss // Senior Systems Admin
>>>>
>>>> e:granwille@namhost.com<ma...@namhost.com>
>>>> m: +264 81 323 1260<tel:+264813231260>
>>>> w:www.namhost.com<https://www.namhost.com/>
>>>>
>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>
>>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>> Namhost Internet Services (Pty) Ltd,
>>>>
>>>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>>>
>>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>>
>>>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>> --
>>>>
>>>> Regards / Groete
>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
>>>> Granwille Strauss // Senior Systems Admin
>>>>
>>>> e:granwille@namhost.com<ma...@namhost.com>
>>>> m: +264 81 323 1260<tel:+264813231260>
>>>> w:www.namhost.com<https://www.namhost.com/>
>>>>
>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>
>>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>> Namhost Internet Services (Pty) Ltd,
>>>>
>>>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>>>
>>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>>
>>>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>> --
>>>>
>>>> Regards / Groete
>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
>>>> Granwille Strauss // Senior Systems Admin
>>>>
>>>> e:granwille@namhost.com<ma...@namhost.com>
>>>> m: +264 81 323 1260<tel:+264813231260>
>>>> w:www.namhost.com<https://www.namhost.com/>
>>>>
>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>
>>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>> Namhost Internet Services (Pty) Ltd,
>>>>
>>>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>>>
>>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>>
>>>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>>
>>> --
>>> Regards / Groete
>>>
>>> <https://www.namhost.com> Granwille Strauss // Senior Systems Admin
>>>
>>> *e:* granwille@namhost.com
>>> *m:* +264 81 323 1260 <tel:+264813231260>
>>> *w:* www.namhost.com <https://www.namhost.com/>
>>>
>>> <https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>
>>> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>
>>> Namhost Internet Services (Pty) Ltd,
>>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>>
>>>
>>> The content of this message is confidential. If you have received it
>>> by mistake, please inform us by email reply and then delete the
>>> message. It is forbidden to copy, forward, or in any way reveal the
>>> contents of this message to anyone without our explicit consent. The
>>> integrity and security of this email cannot be guaranteed over the
>>> Internet. Therefore, the sender will not be held liable for any
>>> damage caused by the message. For our full privacy policy and
>>> disclaimers, please go to https://www.namhost.com/privacy-policy
>>>
>>> Powered by AdSigner
>>> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>> --
>> Regards / Groete
>>
>> <https://www.namhost.com> Granwille Strauss // Senior Systems Admin
>>
>> *e:* granwille@namhost.com
>> *m:* +264 81 323 1260 <tel:+264813231260>
>> *w:* www.namhost.com <https://www.namhost.com/>
>>
>> <https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>
>> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>
>> Namhost Internet Services (Pty) Ltd,
>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>
>>
>> The content of this message is confidential. If you have received it
>> by mistake, please inform us by email reply and then delete the
>> message. It is forbidden to copy, forward, or in any way reveal the
>> contents of this message to anyone without our explicit consent. The
>> integrity and security of this email cannot be guaranteed over the
>> Internet. Therefore, the sender will not be held liable for any
>> damage caused by the message. For our full privacy policy and
>> disclaimers, please go to https://www.namhost.com/privacy-policy
>>
>> Powered by AdSigner
>> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
> --
> Regards / Groete
>
> <https://www.namhost.com> Granwille Strauss // Senior Systems Admin
>
> *e:* granwille@namhost.com
> *m:* +264 81 323 1260 <tel:+264813231260>
> *w:* www.namhost.com <https://www.namhost.com/>
>
> <https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>
> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>
> Namhost Internet Services (Pty) Ltd,
> 24 Black Eagle Rd, Hermanus, 7210, RSA
>
>
> The content of this message is confidential. If you have received it
> by mistake, please inform us by email reply and then delete the
> message. It is forbidden to copy, forward, or in any way reveal the
> contents of this message to anyone without our explicit consent. The
> integrity and security of this email cannot be guaranteed over the
> Internet. Therefore, the sender will not be held liable for any damage
> caused by the message. For our full privacy policy and disclaimers,
> please go to https://www.namhost.com/privacy-policy
>
> Powered by AdSigner
> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
--
Regards / Groete
<https://www.namhost.com> Granwille Strauss // Senior Systems Admin
*e:* granwille@namhost.com
*m:* +264 81 323 1260 <tel:+264813231260>
*w:* www.namhost.com <https://www.namhost.com/>
<https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
Namhost Internet Services (Pty) Ltd,
24 Black Eagle Rd, Hermanus, 7210, RSA
The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It
is forbidden to copy, forward, or in any way reveal the contents of this
message to anyone without our explicit consent. The integrity and
security of this email cannot be guaranteed over the Internet.
Therefore, the sender will not be held liable for any damage caused by
the message. For our full privacy policy and disclaimers, please go to
https://www.namhost.com/privacy-policy
Powered by AdSigner
<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
Re: Recurring Snaps Failing
Posted by Granwille Strauss <gr...@namhost.com.INVALID>.
Hi Guys
I have logged the issue on GITHub:
https://github.com/apache/cloudstack/issues/6836 I would appreciate it
if you can take some time to comment on the issue on GITHub, to help
better explain what the problem is, please. Thank you very much for your
assistance and patience thus far. Hopefully we can get a solution soon.
On 10/18/22 17:04, Granwille Strauss wrote:
>
> Hi
>
> It failed again right after 1 hour, even after I changed the 'wait'
> from 1800 to 3600. Is there not another "wait" setting that may need
> to be increased?
>
> On 10/18/22 15:38, Granwille Strauss wrote:
>>
>> Hi
>>
>> Alright, lets give that a shot. My `wait` in global config was set as
>> 1800 seconds. I have increased it to 3600 seconds, restarted
>> cloudstack-management and started a volume snapshot. Lets see.
>> Attached is a screenshot of all wait settings I have in place, please
>> let me know if I missed anything?
>>
>> On 10/18/22 15:31, Nicolas Vazquez wrote:
>>> Hi,
>>>
>>> As a final test I would suggest increasing the value of the ‘wait’ configuration. Checking the logs, it does not ever reach the point in which the snapshot is copied to the snapshots directory on primary storage and it may be taking too long. Currently, the operation is being timeout after 3600 seconds (2 * wait value?), can you try increasing it? It would require restarting the management server after applying the new value. If this doesn’t help please create a new issue on Github
>>>
>>> Regards,
>>> Nicolas Vazquez
>>>
>>>
>>> From: Granwille Strauss<gr...@namhost.com>
>>> Date: Tuesday, 18 October 2022 at 09:31
>>> To: Wei ZHOU<us...@gmail.com>
>>> Cc: Nicolas Vazquez<Ni...@shapeblue.com>,users@cloudstack.apache.org <us...@cloudstack.apache.org>
>>> Subject: Re: Recurring Snaps Failing
>>>
>>> Hi Guys
>>>
>>> Was there anything else I could do to get this fixed? Should I report this on Github? Also, is it possible to make recurring VM Snapshots and not only Volume Snapshots?
>>> On 10/18/22 11:43, Granwille Strauss wrote:
>>>
>>> Hi Wei
>>>
>>> Correct, yes VM Snapshot is 100% successful, its only volume snapshot that fails and its at the conversion it seems. As requested:
>>> root@HOSTNAME ~ $ virsh domblklist i-2-19-VM
>>> Target Source
>>> ------------------------------------------------------------------------------------------
>>> vda /mnt/81ffca3a-9775-375d-a1c0-9504c0ec3d89/635fd5b5-bf14-4cf4-979c-be8a649d5189
>>> hdc -
>>>
>>> root@HOSTNAME ~ $ qemu-img info /mnt/81ffca3a-9775-375d-a1c0-9504c0ec3d89/635fd5b5-bf14-4cf4-979c-be8a649d5189
>>> qemu-img: Could not open '/mnt/81ffca3a-9775-375d-a1c0-9504c0ec3d89/635fd5b5-bf14-4cf4-979c-be8a649d5189': Failed to get shared "write" lock
>>> Is another process using the image [/mnt/81ffca3a-9775-375d-a1c0-9504c0ec3d89/635fd5b5-bf14-4cf4-979c-be8a649d5189]?
>>> root@athena ~ $
>>>
>>> Seems I cannot get you the details, however would this help?
>>> root@HOSTNAME /mnt/81ffca3a-9775-375d-a1c0-9504c0ec3d89 $ du * -sh
>>> 48G 340cc676-4683-4424-beb7-8b33a5cb3dfe
>>> 42G 4a56546f-4f63-41c2-a314-27619fe5c2dc
>>> 15G 635fd5b5-bf14-4cf4-979c-be8a649d5189
>>> 21G 70ffef79-3aad-43ab-93d1-153fcd34ad15
>>> 26G 777b5149-50c0-4c0d-944b-873050c85419
>>> 20G 867a61e8-4635-40e4-b1b1-c94b2a325b49
>>> 4.8G b21551c6-8276-4e58-841e-4e5a02c0e163
>>> 11G d2c4feca-da5c-44b3-8ecc-40a01fd6c243
>>> 40G f50c0bca-3ae2-48ee-aa9d-0e2d4ec6ba92
>>> 24G f9336d09-0b3d-42d8-9a24-883b022b6712
>>> 8.9G f962cb2c-e03b-4ed2-a035-1870a367aa46
>>> 47G fd573618-885e-48c7-86e6-b019e2311d76
>>> 8.0K KVMHA
>>> 15G snapshots
>>> root@HOSTNAME /mnt/81ffca3a-9775-375d-a1c0-9504c0ec3d89 $
>>>
>>>
>>> I have also added latest agent.log from today.
>>> On 10/18/22 11:07, Wei ZHOU wrote:
>>> Hi,
>>>
>>> You can get the path of qcow2 image by "virsh domblklist i-2-19-VM". The QCOW2 image does not have .qcow2 as a suffix.
>>>
>>> From what I see, a vm snapshot has been created 2022-10-18 09:44:21 +0200, which is good as taking the snapshot started at the same time.
>>> so timed out when converting the vm snapshot to a qcow2 image (ons secondary storage) by "qemu-img convert".
>>>
>>> ps: the agent.log you attached is out-of-date. Please share the agent.log in few minutes after "2022-10-18 09:44:21"
>>>
>>>
>>> -Wei
>>>
>>>
>>>
>>>
>>> On Tue, 18 Oct 2022 at 10:53, Granwille Strauss <gr...@namhost.com>> wrote:
>>>
>>> Hi Wei
>>>
>>> Thank you for the update. Thus far all the snapshot tests I took were volume snaps and majority were when they are online and end up failing. However, there's one VM where it seems to work just fine, whether its online of offline and I am not sure what's different from this VM as all other VM use the same Almalinux template. Its just this one VM where it works as expected.
>>>
>>> In any case, I did a VM snapshot now with memory on my Ubuntu vm while it was online and the VM Snap was successful, no issues there. I ran a volume snap right after on the same ubuntu vm and it failed again as before, see attached screenshot. So VM snapshots seem to work fine, this tells me the issue is most likely when its converted to QCOW2.
>>>
>>> In any case, please see answers to your questions below:
>>>
>>> 1. I am not finding any .qcow2 images on the KVM server to list the QCOW2 size, or am I missing something?
>>> root@HOSTNAME ~ $ find / -type f -iname '*.qcow2'
>>> root@HOSTNAME ~ $
>>> As requested, here's a snapshot list of the current vm I am testing on. Please let me know if you needed something specific?
>>> root@HOSTNAME ~ $ virsh snapshot-list i-2-19-VM
>>> Name Creation Time State
>>> -----------------------------------------------------------------------------------
>>> 0843e6b1-cb12-495c-8d5f-35c9a3338db7 2022-10-17 18:41:08 +0200 disk-snapshot
>>> 2e77acb7-1be4-4417-858e-a1d0890f44f0 2022-10-17 19:57:03 +0200 disk-snapshot
>>> 68476e2a-aec8-4637-9c4b-2dc49d79bd82 2022-10-18 09:44:21 +0200 disk-snapshot
>>> c110058b-4661-42c0-b648-a54f4ba4beb7 2022-10-17 22:34:21 +0200 disk-snapshot
>>> i-2-19-VM_VS_20221018072924 2022-10-18 09:29:25 +0200 running
>>>
>>> 2. Attached is a txt file from yesterdays agent.log. This is all entries that happened between CopyCommand and CopyCmdAnswer. There are zero errors, as you will see.
>>>
>>> 3. The 'backup.snapshot.wait' setting is currently at 21600 seconds = 6 hours, should this still be increased? Also, not sure if 'copy.volume.wait' is relevant, but this is currently at 10800 seconds.
>>> On 10/18/22 09:30, Wei ZHOU wrote:
>>> Hi Granwille,
>>>
>>> The process of volume snapshot on running vms and stop vms are different. For running vms, cloudstack will first take a vm snapshot and convert the vm snapshot into a qcow2 image.
>>> It is normal that snapshotting a running vm takes longer time than a stopped vm, because
>>> (1) there is an extra step to take a vm snapshot. Depending on the memory size, the step might take a few minutes.
>>> (3) vm snapshot is saved in QCOW2 image (it is called Internal snapshots). When taking a vm snapshot, the QCOW2 image will enlarge. The image will not shrink when the vm snapshot is removed.
>>> If you have taken vm snapshot (or volume snapshot on running vms) multiple times, The QCOW2 image might be very big. The whole process (including vm snapshotting and qemu-img convert) might exceed the timeout limit (3600 seconds = 1 hour).
>>>
>>> can you please
>>> (1) check the QCOW2 image size and list the snapshot of QCOW2 image (qemu-img info), list vm snapshot of the vm (virsh snapshot-list)
>>> (2) check the logs between CopyCommand and CopyCmdAnswer in agent.log
>>> (3) increase backup.snapshot.wait in global configuration and retry ("wait" in "CopyCommand" should be changed accordingly) ?
>>>
>>>
>>> -Wei
>>>
>>>
>>>
>>> On Tue, 18 Oct 2022 at 00:39, Granwille Strauss <gr...@namhost.com>> wrote:
>>>
>>> Hi Guys
>>>
>>> I tested another Linux VM Almalinux VM, turned off the VM and made a successful snap. I turned it back online again and run another successful snap again. Now I run a snap on my Ubuntu VM and it failed. So just to be sure it wasn't just that one specific Almalinux VM that was working, I ran another snap on another Almalinux VM while it was online and too also failed. So I am not sure why recurring snaps is only working for this one VM and not the rest.
>>>
>>> Should I log this on GITHUB?
>>> On 10/17/22 21:55, Granwille Strauss wrote:
>>>
>>> Sorry, seems my attachments were too big, you can download via Wetransfer:https://we.tl/t-9hUekbj86S
>>>
>>> And yes, I have the kvm.snapshot.enabled enabled in global settings too, for live snapping.
>>> On 10/17/22 21:51, Granwille Strauss wrote:
>>>
>>> So it seems when VM is powered off, snap is successful. See attached logs for the successful snap, when VM is offline.
>>> On 10/17/22 21:38, Granwille Strauss wrote:
>>>
>>> Here's the last 10 minutes before the error is triggered in the logs, this was the test run before switching off VM. That test is still running and will provide its logs shortly, however, I am confident, it will show same results.
>>> On 10/17/22 21:22, Granwille Strauss wrote:
>>>
>>> Hi Wei
>>>
>>> Please see attached. Please keep in mind, this worked 100% a few days ago when I tested on VM recurring snaps and it was perfect then. So I enabled recurring snaps for all 10 linux VMs I host. And I set them all to daily at 2AM.
>>> They failed after the first day. So I figured maybe too many concurrent jobs running at the same time.
>>>
>>> So I split them all to once weekly at different days at different hours. And still it failed. And now when doing it manually it fails too. But for some reason the snap ends up in /primary location is just returns an error for some reason.
>>> On 10/17/22 21:13, Wei ZHOU wrote:
>>>
>>> Hi,
>>>
>>>
>>>
>>> What is the kvm host os and version ?
>>>
>>>
>>>
>>> -Wei
>>>
>>>
>>>
>>> On Monday, 17 October 2022, Granwille Strauss<gr...@namhost.com.invalid>
>>>
>>> wrote:
>>>
>>>
>>>
>>> The error seems to happen exactly at 1 hour intervals. All of them have
>>>
>>> failed exactly at one hour. Also, these are all volume snapshots that I am
>>>
>>> taking, so the logs I sent you so far are all in this regard. They are not
>>>
>>> VM snaphots, nor are they powered off. I am running a new volume snap shot
>>>
>>> again, this time VM powered off lets see.
>>>
>>>
>>>
>>> I have also rebooted my entire KVM server and my management server. So far
>>>
>>> zero luck.
>>>
>>> On 10/17/22 20:26, Nicolas Vazquez wrote:
>>>
>>>
>>>
>>> Can you try taking a manual volume snapshot and share both mgmt and agent logs please?
>>>
>>>
>>>
>>> Regards,
>>>
>>> Nicolas Vazquez
>>>
>>>
>>>
>>>
>>>
>>> From: Granwille Strauss<gr...@namhost.com> <gr...@namhost.com>
>>>
>>> Date: Monday, 17 October 2022 at 13:15
>>>
>>> To:users@cloudstack.apache.org<ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org>
>>>
>>> Cc: Nicolas Vazquez<Ni...@shapeblue.com> <Ni...@shapeblue.com>
>>>
>>> Subject: Re: Recurring Snaps Failing
>>>
>>>
>>>
>>> Hi
>>>
>>>
>>>
>>> I also see another issue with snaps, I deleted all the failed snaps via the cloud management UI. But to my surprise, they're all still at /mnt/primary/snapshots/ see attached screenshot. Something odd is going on with snapshots.
>>>
>>> On 10/17/22 17:35, Granwille Strauss wrote:
>>>
>>>
>>>
>>> Hi
>>>
>>>
>>>
>>> I am not 100% if I enabled debug correctly, I followed thishttps://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+agent+debug and ran
>>>
>>> sed -i 's/INFO/DEBUG/g' /etc/cloudstack/agent/log4j-cloud.xml
>>>
>>>
>>>
>>> Attached are the two log files. Same error. I reboot SSVM & vRouter prior no luck.
>>>
>>> On 10/17/22 16:00, Granwille Strauss wrote:
>>>
>>>
>>>
>>> Hi
>>>
>>>
>>>
>>> No KVM is on its own server node and management on a different server node. However, management is is primary and secondary storage via NFS. I will enable DEBUG and re-run manual snap to replicate the error.
>>>
>>> On 10/17/22 15:42, Nicolas Vazquez wrote:
>>>
>>> Thanks, are the KVM agent and management server running on the same host? Can you increase the log level on the KVM agent logs to DEBUG and retry?
>>>
>>>
>>>
>>> Regards,
>>>
>>> Nicolas Vazquez
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> From: Granwille Strauss<gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com>
>>>
>>> Date: Monday, 17 October 2022 at 10:04
>>>
>>> To:users@cloudstack.apache.org<ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org>
>>>
>>> Cc: Nicolas Vazquez<Ni...@shapeblue.com> <Ni...@shapeblue.com> <Ni...@shapeblue.com>
>>>
>>> Subject: Re: Recurring Snaps Failing
>>>
>>>
>>>
>>> Please find attached TXT
>>>
>>> On 10/17/22 14:54, Nicolas Vazquez wrote:
>>>
>>>
>>>
>>> Ok, seems like the command is not reaching the KVM host, can you share the full management server logs for job-796?
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Regards,
>>>
>>>
>>>
>>> Nicolas Vazquez
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> From: Granwille Strauss<gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com>
>>>
>>>
>>>
>>> Date: Monday, 17 October 2022 at 09:32
>>>
>>>
>>>
>>> To:users@cloudstack.apache.org<ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org>
>>>
>>>
>>>
>>> Cc: Nicolas Vazquez<Ni...@shapeblue.com> <Ni...@shapeblue.com> <Ni...@shapeblue.com>
>>>
>>>
>>>
>>> Subject: Re: Recurring Snaps Failing
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Hi
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> This is all agent threw during the snap process nothing else:
>>>
>>>
>>>
>>> 2022-10-17 13:39:12,375 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:a864f9f6) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
>>>
>>>
>>>
>>> 2022-10-17 13:40:13,276 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:445216f3) Trying to fetch storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297 from libvirt
>>>
>>>
>>>
>>> 2022-10-17 13:40:13,289 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:445216f3) Asking libvirt to refresh storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297
>>>
>>>
>>>
>>> 2022-10-17 13:40:13,378 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:445216f3) Trying to fetch storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89 from libvirt
>>>
>>>
>>>
>>> 2022-10-17 13:40:13,393 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:445216f3) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
>>>
>>>
>>>
>>> 2022-10-17 13:41:13,950 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:87ecfee0) Trying to fetch storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297 from libvirt
>>>
>>>
>>>
>>> 2022-10-17 13:41:13,964 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:87ecfee0) Asking libvirt to refresh storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297
>>>
>>>
>>>
>>> 2022-10-17 13:41:14,054 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:87ecfee0) Trying to fetch storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89 from libvirt
>>>
>>>
>>>
>>> 2022-10-17 13:41:14,067 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:87ecfee0) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
>>>
>>>
>>>
>>> 2022-10-17 13:41:57,753 WARN [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-3:null) (logid:d91c35e5) Expected 1 answers while executing SetMonitorServiceCommand but received 3
>>>
>>>
>>>
>>> 2022-10-17 13:41:57,938 INFO [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:null) (logid:1549c9a1) Fetching health check result for SEVERIPADDRESS and executing fresh checks: false
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On 10/17/22 14:29, Nicolas Vazquez wrote:
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Hi Groete,
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Can you share the CloudStack agent logs from the KVM host?
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Regards,
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Nicolas Vazquez
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> From: Granwille Strauss<gr...@namhost.com.INVALID> <gr...@namhost.com.INVALID> <gr...@namhost.com.INVALID> <gr...@namhost.com.INVALID> <gr...@namhost.com.INVALID>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Date: Monday, 17 October 2022 at 08:56
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> To:users@cloudstack.apache.org<ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Subject: Re: Recurring Snaps Failing
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Hi
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> I ran a few more tests and can now finally make sense of the failure:
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> 2022-10-17 13:39:08,556 DEBUG [o.a.c.s.d.d.CloudStackPrimaryDataStoreDriverImpl] (Work-Job-Executor-1:ctx-93cb161b job-806/job-807 ctx-3f9ccf45) (logid:8fd10b20) Failed to take snapshot: 23
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> 2022-10-17 13:39:08,558 DEBUG [o.a.c.s.s.SnapshotServiceImpl] (Work-Job-Executor-1:ctx-93cb161b job-806/job-807 ctx-3f9ccf45) (logid:8fd10b20) create snapshot Test001 failed: com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:1, com.cloud.exception.OperationTimedoutException: Commands 7160441932542378205 to Host 1 timed out after 3600
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Seems to be a time out error. But there's no firewalls on the KVM host and NFS server to prevent connections and it worked fine before and randomly just stopped.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> So I still have no idea where to solve this as all other necessary settings are in place. Any ideas?
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On 10/17/22 08:51, Granwille Strauss wrote:
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Hi Guys
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> I enabled recurring snap shots for my Linux VMs and they all seem to be failing with the following error:
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> root@hostname ~ $ grep error /var/log/cloudstack/management/management-server.log
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> 2022-10-17 03:01:29,474 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-27:ctx-99d7d702 job-796 ctx-f26a81fe) (logid:f9a61b00) Failed to create snapshot due to an internal error creating snapshot for volume 635fd5b5-bf14-4cf4-979c-be8a649d5189
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> 2022-10-17 03:01:29,476 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-27:ctx-99d7d702 job-796) (logid:f9a61b00) Complete async job-796, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 635fd5b5-bf14-4cf4-979c-be8a649d5189"}
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> 2022-10-17 03:01:29,478 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-26:ctx-c7865967 job-794 ctx-0700afb0) (logid:5541113c) Failed to create snapshot due to an internal error creating snapshot for volume 70ffef79-3aad-43ab-93d1-153fcd34ad15
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> 2022-10-17 03:01:29,480 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-26:ctx-c7865967 job-794) (logid:5541113c) Complete async job-794, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 70ffef79-3aad-43ab-93d1-153fcd34ad15"}
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> All I am getting out of this is internal error and nothing else. I have checked other logs and I am not quite sure where else to look.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Any assistance in this regard will be appreciated.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Regards / Groete
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Granwille Strauss // Senior Systems Admin
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> e:granwille@namhost.com<ma...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> m: +264 81 323 1260<tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> w:www.namhost.com<http://www.namhost.com><http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com><http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com><http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Namhost Internet Services (Pty) Ltd,
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g><https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Regards / Groete
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Granwille Strauss // Senior Systems Admin
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> e:granwille@namhost.com<ma...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> m: +264 81 323 1260<tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> w:www.namhost.com<http://www.namhost.com><http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com><http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com><http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Namhost Internet Services (Pty) Ltd,
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g><https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Regards / Groete
>>>
>>>
>>>
>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
>>>
>>>
>>>
>>> Granwille Strauss // Senior Systems Admin
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> e:granwille@namhost.com<ma...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com>
>>>
>>>
>>>
>>> m: +264 81 323 1260<tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260>
>>>
>>>
>>>
>>> w:www.namhost.com<http://www.namhost.com><http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>
>>>
>>>
>>> Namhost Internet Services (Pty) Ltd,
>>>
>>>
>>>
>>>
>>>
>>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g><https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>
>>>
>>>
>>>
>>>
>>> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>>
>>> Regards / Groete
>>>
>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
>>>
>>> Granwille Strauss // Senior Systems Admin
>>>
>>>
>>>
>>> e:granwille@namhost.com<ma...@namhost.com> <gr...@namhost.com>
>>>
>>> m: +264 81 323 1260<tel:+264813231260><tel:+264813231260>
>>>
>>> w:www.namhost.com<http://www.namhost.com><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
>>>
>>>
>>>
>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>
>>>
>>>
>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>
>>> Namhost Internet Services (Pty) Ltd,
>>>
>>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g><https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>>
>>>
>>>
>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>
>>>
>>>
>>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>
>>> --
>>>
>>>
>>>
>>> Regards / Groete
>>>
>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
>>>
>>> Granwille Strauss // Senior Systems Admin
>>>
>>>
>>>
>>> e:granwille@namhost.com<ma...@namhost.com> <gr...@namhost.com>
>>>
>>> m: +264 81 323 1260<tel:+264813231260><tel:+264813231260>
>>>
>>> w:www.namhost.com<http://www.namhost.com><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
>>>
>>>
>>>
>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>
>>>
>>>
>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>
>>> Namhost Internet Services (Pty) Ltd,
>>>
>>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g><https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>>
>>>
>>>
>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and s
>>>
>>>
>>>
>>> --
>>>
>>> Regards / Groete
>>>
>>>
>>>
>>> <https://www.namhost.com><https://www.namhost.com> Granwille Strauss // Senior Systems Admin
>>>
>>>
>>>
>>> *e:*granwille@namhost.com<ma...@namhost.com>
>>>
>>> *m:* +264 81 323 1260 <+264813231260>
>>>
>>> *w:*www.namhost.com<http://www.namhost.com>
>>>
>>>
>>>
>>> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost>
>>>
>>> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/>
>>>
>>> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos>
>>>
>>> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>
>>>
>>>
>>>
>>>
>>> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>
>>>
>>>
>>> Namhost Internet Services (Pty) Ltd,
>>>
>>>
>>>
>>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g><https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> The content of this message is confidential. If you have received it by
>>>
>>> mistake, please inform us by email reply and then delete the message. It is
>>>
>>> forbidden to copy, forward, or in any way reveal the contents of this
>>>
>>> message to anyone without our explicit consent. The integrity and security
>>>
>>> of this email cannot be guaranteed over the Internet. Therefore, the sender
>>>
>>> will not be held liable for any damage caused by the message. For our full
>>>
>>> privacy policy and disclaimers, please go tohttps://www.namhost.com/
>>>
>>> privacy-policy
>>>
>>>
>>>
>>> [image: Powered by AdSigner]
>>>
>>> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>
>>>
>>> --
>>> Regards / Groete
>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
>>> Granwille Strauss // Senior Systems Admin
>>>
>>> e:granwille@namhost.com<ma...@namhost.com>
>>> m: +264 81 323 1260<tel:+264813231260>
>>> w:www.namhost.com<https://www.namhost.com/>
>>>
>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>
>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>> Namhost Internet Services (Pty) Ltd,
>>>
>>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>>
>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>
>>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>> --
>>> Regards / Groete
>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
>>> Granwille Strauss // Senior Systems Admin
>>>
>>> e:granwille@namhost.com<ma...@namhost.com>
>>> m: +264 81 323 1260<tel:+264813231260>
>>> w:www.namhost.com<https://www.namhost.com/>
>>>
>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>
>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>> Namhost Internet Services (Pty) Ltd,
>>>
>>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>>
>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>
>>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>> --
>>> Regards / Groete
>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
>>> Granwille Strauss // Senior Systems Admin
>>>
>>> e:granwille@namhost.com<ma...@namhost.com>
>>> m: +264 81 323 1260<tel:+264813231260>
>>> w:www.namhost.com<https://www.namhost.com/>
>>>
>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>
>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>> Namhost Internet Services (Pty) Ltd,
>>>
>>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>>
>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>
>>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>> --
>>> Regards / Groete
>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
>>> Granwille Strauss // Senior Systems Admin
>>>
>>> e:granwille@namhost.com<ma...@namhost.com>
>>> m: +264 81 323 1260<tel:+264813231260>
>>> w:www.namhost.com<https://www.namhost.com/>
>>>
>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>
>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>> Namhost Internet Services (Pty) Ltd,
>>>
>>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>>
>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>
>>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>> --
>>> Regards / Groete
>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
>>> Granwille Strauss // Senior Systems Admin
>>>
>>> e:granwille@namhost.com<ma...@namhost.com>
>>> m: +264 81 323 1260<tel:+264813231260>
>>> w:www.namhost.com<https://www.namhost.com/>
>>>
>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>
>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>> Namhost Internet Services (Pty) Ltd,
>>>
>>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>>
>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>
>>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>> --
>>> Regards / Groete
>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
>>> Granwille Strauss // Senior Systems Admin
>>>
>>> e:granwille@namhost.com<ma...@namhost.com>
>>> m: +264 81 323 1260<tel:+264813231260>
>>> w:www.namhost.com<https://www.namhost.com/>
>>>
>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>
>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>> Namhost Internet Services (Pty) Ltd,
>>>
>>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>>
>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>
>>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>> --
>>>
>>> Regards / Groete
>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
>>> Granwille Strauss // Senior Systems Admin
>>>
>>> e:granwille@namhost.com<ma...@namhost.com>
>>> m: +264 81 323 1260<tel:+264813231260>
>>> w:www.namhost.com<https://www.namhost.com/>
>>>
>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>
>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>> Namhost Internet Services (Pty) Ltd,
>>>
>>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>>
>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>
>>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>> --
>>>
>>> Regards / Groete
>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
>>> Granwille Strauss // Senior Systems Admin
>>>
>>> e:granwille@namhost.com<ma...@namhost.com>
>>> m: +264 81 323 1260<tel:+264813231260>
>>> w:www.namhost.com<https://www.namhost.com/>
>>>
>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>
>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>> Namhost Internet Services (Pty) Ltd,
>>>
>>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>>
>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>
>>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>
>> --
>> Regards / Groete
>>
>> <https://www.namhost.com> Granwille Strauss // Senior Systems Admin
>>
>> *e:* granwille@namhost.com
>> *m:* +264 81 323 1260 <tel:+264813231260>
>> *w:* www.namhost.com <https://www.namhost.com/>
>>
>> <https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>
>> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>
>> Namhost Internet Services (Pty) Ltd,
>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>
>>
>> The content of this message is confidential. If you have received it
>> by mistake, please inform us by email reply and then delete the
>> message. It is forbidden to copy, forward, or in any way reveal the
>> contents of this message to anyone without our explicit consent. The
>> integrity and security of this email cannot be guaranteed over the
>> Internet. Therefore, the sender will not be held liable for any
>> damage caused by the message. For our full privacy policy and
>> disclaimers, please go to https://www.namhost.com/privacy-policy
>>
>> Powered by AdSigner
>> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
> --
> Regards / Groete
>
> <https://www.namhost.com> Granwille Strauss // Senior Systems Admin
>
> *e:* granwille@namhost.com
> *m:* +264 81 323 1260 <tel:+264813231260>
> *w:* www.namhost.com <https://www.namhost.com/>
>
> <https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>
> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>
> Namhost Internet Services (Pty) Ltd,
> 24 Black Eagle Rd, Hermanus, 7210, RSA
>
>
> The content of this message is confidential. If you have received it
> by mistake, please inform us by email reply and then delete the
> message. It is forbidden to copy, forward, or in any way reveal the
> contents of this message to anyone without our explicit consent. The
> integrity and security of this email cannot be guaranteed over the
> Internet. Therefore, the sender will not be held liable for any damage
> caused by the message. For our full privacy policy and disclaimers,
> please go to https://www.namhost.com/privacy-policy
>
> Powered by AdSigner
> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
--
Regards / Groete
<https://www.namhost.com> Granwille Strauss // Senior Systems Admin
*e:* granwille@namhost.com
*m:* +264 81 323 1260 <tel:+264813231260>
*w:* www.namhost.com <https://www.namhost.com/>
<https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
Namhost Internet Services (Pty) Ltd,
24 Black Eagle Rd, Hermanus, 7210, RSA
The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It
is forbidden to copy, forward, or in any way reveal the contents of this
message to anyone without our explicit consent. The integrity and
security of this email cannot be guaranteed over the Internet.
Therefore, the sender will not be held liable for any damage caused by
the message. For our full privacy policy and disclaimers, please go to
https://www.namhost.com/privacy-policy
Powered by AdSigner
<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
Re: Recurring Snaps Failing
Posted by Granwille Strauss <gr...@namhost.com.INVALID>.
Hi
It failed again right after 1 hour, even after I changed the 'wait' from
1800 to 3600. Is there not another "wait" setting that may need to be
increased?
On 10/18/22 15:38, Granwille Strauss wrote:
>
> Hi
>
> Alright, lets give that a shot. My `wait` in global config was set as
> 1800 seconds. I have increased it to 3600 seconds, restarted
> cloudstack-management and started a volume snapshot. Lets see.
> Attached is a screenshot of all wait settings I have in place, please
> let me know if I missed anything?
>
> On 10/18/22 15:31, Nicolas Vazquez wrote:
>> Hi,
>>
>> As a final test I would suggest increasing the value of the ‘wait’ configuration. Checking the logs, it does not ever reach the point in which the snapshot is copied to the snapshots directory on primary storage and it may be taking too long. Currently, the operation is being timeout after 3600 seconds (2 * wait value?), can you try increasing it? It would require restarting the management server after applying the new value. If this doesn’t help please create a new issue on Github
>>
>> Regards,
>> Nicolas Vazquez
>>
>>
>> From: Granwille Strauss<gr...@namhost.com>
>> Date: Tuesday, 18 October 2022 at 09:31
>> To: Wei ZHOU<us...@gmail.com>
>> Cc: Nicolas Vazquez<Ni...@shapeblue.com>,users@cloudstack.apache.org <us...@cloudstack.apache.org>
>> Subject: Re: Recurring Snaps Failing
>>
>> Hi Guys
>>
>> Was there anything else I could do to get this fixed? Should I report this on Github? Also, is it possible to make recurring VM Snapshots and not only Volume Snapshots?
>> On 10/18/22 11:43, Granwille Strauss wrote:
>>
>> Hi Wei
>>
>> Correct, yes VM Snapshot is 100% successful, its only volume snapshot that fails and its at the conversion it seems. As requested:
>> root@HOSTNAME ~ $ virsh domblklist i-2-19-VM
>> Target Source
>> ------------------------------------------------------------------------------------------
>> vda /mnt/81ffca3a-9775-375d-a1c0-9504c0ec3d89/635fd5b5-bf14-4cf4-979c-be8a649d5189
>> hdc -
>>
>> root@HOSTNAME ~ $ qemu-img info /mnt/81ffca3a-9775-375d-a1c0-9504c0ec3d89/635fd5b5-bf14-4cf4-979c-be8a649d5189
>> qemu-img: Could not open '/mnt/81ffca3a-9775-375d-a1c0-9504c0ec3d89/635fd5b5-bf14-4cf4-979c-be8a649d5189': Failed to get shared "write" lock
>> Is another process using the image [/mnt/81ffca3a-9775-375d-a1c0-9504c0ec3d89/635fd5b5-bf14-4cf4-979c-be8a649d5189]?
>> root@athena ~ $
>>
>> Seems I cannot get you the details, however would this help?
>> root@HOSTNAME /mnt/81ffca3a-9775-375d-a1c0-9504c0ec3d89 $ du * -sh
>> 48G 340cc676-4683-4424-beb7-8b33a5cb3dfe
>> 42G 4a56546f-4f63-41c2-a314-27619fe5c2dc
>> 15G 635fd5b5-bf14-4cf4-979c-be8a649d5189
>> 21G 70ffef79-3aad-43ab-93d1-153fcd34ad15
>> 26G 777b5149-50c0-4c0d-944b-873050c85419
>> 20G 867a61e8-4635-40e4-b1b1-c94b2a325b49
>> 4.8G b21551c6-8276-4e58-841e-4e5a02c0e163
>> 11G d2c4feca-da5c-44b3-8ecc-40a01fd6c243
>> 40G f50c0bca-3ae2-48ee-aa9d-0e2d4ec6ba92
>> 24G f9336d09-0b3d-42d8-9a24-883b022b6712
>> 8.9G f962cb2c-e03b-4ed2-a035-1870a367aa46
>> 47G fd573618-885e-48c7-86e6-b019e2311d76
>> 8.0K KVMHA
>> 15G snapshots
>> root@HOSTNAME /mnt/81ffca3a-9775-375d-a1c0-9504c0ec3d89 $
>>
>>
>> I have also added latest agent.log from today.
>> On 10/18/22 11:07, Wei ZHOU wrote:
>> Hi,
>>
>> You can get the path of qcow2 image by "virsh domblklist i-2-19-VM". The QCOW2 image does not have .qcow2 as a suffix.
>>
>> From what I see, a vm snapshot has been created 2022-10-18 09:44:21 +0200, which is good as taking the snapshot started at the same time.
>> so timed out when converting the vm snapshot to a qcow2 image (ons secondary storage) by "qemu-img convert".
>>
>> ps: the agent.log you attached is out-of-date. Please share the agent.log in few minutes after "2022-10-18 09:44:21"
>>
>>
>> -Wei
>>
>>
>>
>>
>> On Tue, 18 Oct 2022 at 10:53, Granwille Strauss <gr...@namhost.com>> wrote:
>>
>> Hi Wei
>>
>> Thank you for the update. Thus far all the snapshot tests I took were volume snaps and majority were when they are online and end up failing. However, there's one VM where it seems to work just fine, whether its online of offline and I am not sure what's different from this VM as all other VM use the same Almalinux template. Its just this one VM where it works as expected.
>>
>> In any case, I did a VM snapshot now with memory on my Ubuntu vm while it was online and the VM Snap was successful, no issues there. I ran a volume snap right after on the same ubuntu vm and it failed again as before, see attached screenshot. So VM snapshots seem to work fine, this tells me the issue is most likely when its converted to QCOW2.
>>
>> In any case, please see answers to your questions below:
>>
>> 1. I am not finding any .qcow2 images on the KVM server to list the QCOW2 size, or am I missing something?
>> root@HOSTNAME ~ $ find / -type f -iname '*.qcow2'
>> root@HOSTNAME ~ $
>> As requested, here's a snapshot list of the current vm I am testing on. Please let me know if you needed something specific?
>> root@HOSTNAME ~ $ virsh snapshot-list i-2-19-VM
>> Name Creation Time State
>> -----------------------------------------------------------------------------------
>> 0843e6b1-cb12-495c-8d5f-35c9a3338db7 2022-10-17 18:41:08 +0200 disk-snapshot
>> 2e77acb7-1be4-4417-858e-a1d0890f44f0 2022-10-17 19:57:03 +0200 disk-snapshot
>> 68476e2a-aec8-4637-9c4b-2dc49d79bd82 2022-10-18 09:44:21 +0200 disk-snapshot
>> c110058b-4661-42c0-b648-a54f4ba4beb7 2022-10-17 22:34:21 +0200 disk-snapshot
>> i-2-19-VM_VS_20221018072924 2022-10-18 09:29:25 +0200 running
>>
>> 2. Attached is a txt file from yesterdays agent.log. This is all entries that happened between CopyCommand and CopyCmdAnswer. There are zero errors, as you will see.
>>
>> 3. The 'backup.snapshot.wait' setting is currently at 21600 seconds = 6 hours, should this still be increased? Also, not sure if 'copy.volume.wait' is relevant, but this is currently at 10800 seconds.
>> On 10/18/22 09:30, Wei ZHOU wrote:
>> Hi Granwille,
>>
>> The process of volume snapshot on running vms and stop vms are different. For running vms, cloudstack will first take a vm snapshot and convert the vm snapshot into a qcow2 image.
>> It is normal that snapshotting a running vm takes longer time than a stopped vm, because
>> (1) there is an extra step to take a vm snapshot. Depending on the memory size, the step might take a few minutes.
>> (3) vm snapshot is saved in QCOW2 image (it is called Internal snapshots). When taking a vm snapshot, the QCOW2 image will enlarge. The image will not shrink when the vm snapshot is removed.
>> If you have taken vm snapshot (or volume snapshot on running vms) multiple times, The QCOW2 image might be very big. The whole process (including vm snapshotting and qemu-img convert) might exceed the timeout limit (3600 seconds = 1 hour).
>>
>> can you please
>> (1) check the QCOW2 image size and list the snapshot of QCOW2 image (qemu-img info), list vm snapshot of the vm (virsh snapshot-list)
>> (2) check the logs between CopyCommand and CopyCmdAnswer in agent.log
>> (3) increase backup.snapshot.wait in global configuration and retry ("wait" in "CopyCommand" should be changed accordingly) ?
>>
>>
>> -Wei
>>
>>
>>
>> On Tue, 18 Oct 2022 at 00:39, Granwille Strauss <gr...@namhost.com>> wrote:
>>
>> Hi Guys
>>
>> I tested another Linux VM Almalinux VM, turned off the VM and made a successful snap. I turned it back online again and run another successful snap again. Now I run a snap on my Ubuntu VM and it failed. So just to be sure it wasn't just that one specific Almalinux VM that was working, I ran another snap on another Almalinux VM while it was online and too also failed. So I am not sure why recurring snaps is only working for this one VM and not the rest.
>>
>> Should I log this on GITHUB?
>> On 10/17/22 21:55, Granwille Strauss wrote:
>>
>> Sorry, seems my attachments were too big, you can download via Wetransfer:https://we.tl/t-9hUekbj86S
>>
>> And yes, I have the kvm.snapshot.enabled enabled in global settings too, for live snapping.
>> On 10/17/22 21:51, Granwille Strauss wrote:
>>
>> So it seems when VM is powered off, snap is successful. See attached logs for the successful snap, when VM is offline.
>> On 10/17/22 21:38, Granwille Strauss wrote:
>>
>> Here's the last 10 minutes before the error is triggered in the logs, this was the test run before switching off VM. That test is still running and will provide its logs shortly, however, I am confident, it will show same results.
>> On 10/17/22 21:22, Granwille Strauss wrote:
>>
>> Hi Wei
>>
>> Please see attached. Please keep in mind, this worked 100% a few days ago when I tested on VM recurring snaps and it was perfect then. So I enabled recurring snaps for all 10 linux VMs I host. And I set them all to daily at 2AM.
>> They failed after the first day. So I figured maybe too many concurrent jobs running at the same time.
>>
>> So I split them all to once weekly at different days at different hours. And still it failed. And now when doing it manually it fails too. But for some reason the snap ends up in /primary location is just returns an error for some reason.
>> On 10/17/22 21:13, Wei ZHOU wrote:
>>
>> Hi,
>>
>>
>>
>> What is the kvm host os and version ?
>>
>>
>>
>> -Wei
>>
>>
>>
>> On Monday, 17 October 2022, Granwille Strauss<gr...@namhost.com.invalid>
>>
>> wrote:
>>
>>
>>
>> The error seems to happen exactly at 1 hour intervals. All of them have
>>
>> failed exactly at one hour. Also, these are all volume snapshots that I am
>>
>> taking, so the logs I sent you so far are all in this regard. They are not
>>
>> VM snaphots, nor are they powered off. I am running a new volume snap shot
>>
>> again, this time VM powered off lets see.
>>
>>
>>
>> I have also rebooted my entire KVM server and my management server. So far
>>
>> zero luck.
>>
>> On 10/17/22 20:26, Nicolas Vazquez wrote:
>>
>>
>>
>> Can you try taking a manual volume snapshot and share both mgmt and agent logs please?
>>
>>
>>
>> Regards,
>>
>> Nicolas Vazquez
>>
>>
>>
>>
>>
>> From: Granwille Strauss<gr...@namhost.com> <gr...@namhost.com>
>>
>> Date: Monday, 17 October 2022 at 13:15
>>
>> To:users@cloudstack.apache.org<ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org>
>>
>> Cc: Nicolas Vazquez<Ni...@shapeblue.com> <Ni...@shapeblue.com>
>>
>> Subject: Re: Recurring Snaps Failing
>>
>>
>>
>> Hi
>>
>>
>>
>> I also see another issue with snaps, I deleted all the failed snaps via the cloud management UI. But to my surprise, they're all still at /mnt/primary/snapshots/ see attached screenshot. Something odd is going on with snapshots.
>>
>> On 10/17/22 17:35, Granwille Strauss wrote:
>>
>>
>>
>> Hi
>>
>>
>>
>> I am not 100% if I enabled debug correctly, I followed thishttps://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+agent+debug and ran
>>
>> sed -i 's/INFO/DEBUG/g' /etc/cloudstack/agent/log4j-cloud.xml
>>
>>
>>
>> Attached are the two log files. Same error. I reboot SSVM & vRouter prior no luck.
>>
>> On 10/17/22 16:00, Granwille Strauss wrote:
>>
>>
>>
>> Hi
>>
>>
>>
>> No KVM is on its own server node and management on a different server node. However, management is is primary and secondary storage via NFS. I will enable DEBUG and re-run manual snap to replicate the error.
>>
>> On 10/17/22 15:42, Nicolas Vazquez wrote:
>>
>> Thanks, are the KVM agent and management server running on the same host? Can you increase the log level on the KVM agent logs to DEBUG and retry?
>>
>>
>>
>> Regards,
>>
>> Nicolas Vazquez
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> From: Granwille Strauss<gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com>
>>
>> Date: Monday, 17 October 2022 at 10:04
>>
>> To:users@cloudstack.apache.org<ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org>
>>
>> Cc: Nicolas Vazquez<Ni...@shapeblue.com> <Ni...@shapeblue.com> <Ni...@shapeblue.com>
>>
>> Subject: Re: Recurring Snaps Failing
>>
>>
>>
>> Please find attached TXT
>>
>> On 10/17/22 14:54, Nicolas Vazquez wrote:
>>
>>
>>
>> Ok, seems like the command is not reaching the KVM host, can you share the full management server logs for job-796?
>>
>>
>>
>>
>>
>>
>>
>> Regards,
>>
>>
>>
>> Nicolas Vazquez
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> From: Granwille Strauss<gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com>
>>
>>
>>
>> Date: Monday, 17 October 2022 at 09:32
>>
>>
>>
>> To:users@cloudstack.apache.org<ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org>
>>
>>
>>
>> Cc: Nicolas Vazquez<Ni...@shapeblue.com> <Ni...@shapeblue.com> <Ni...@shapeblue.com>
>>
>>
>>
>> Subject: Re: Recurring Snaps Failing
>>
>>
>>
>>
>>
>>
>>
>> Hi
>>
>>
>>
>>
>>
>>
>>
>> This is all agent threw during the snap process nothing else:
>>
>>
>>
>> 2022-10-17 13:39:12,375 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:a864f9f6) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
>>
>>
>>
>> 2022-10-17 13:40:13,276 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:445216f3) Trying to fetch storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297 from libvirt
>>
>>
>>
>> 2022-10-17 13:40:13,289 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:445216f3) Asking libvirt to refresh storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297
>>
>>
>>
>> 2022-10-17 13:40:13,378 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:445216f3) Trying to fetch storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89 from libvirt
>>
>>
>>
>> 2022-10-17 13:40:13,393 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:445216f3) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
>>
>>
>>
>> 2022-10-17 13:41:13,950 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:87ecfee0) Trying to fetch storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297 from libvirt
>>
>>
>>
>> 2022-10-17 13:41:13,964 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:87ecfee0) Asking libvirt to refresh storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297
>>
>>
>>
>> 2022-10-17 13:41:14,054 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:87ecfee0) Trying to fetch storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89 from libvirt
>>
>>
>>
>> 2022-10-17 13:41:14,067 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:87ecfee0) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
>>
>>
>>
>> 2022-10-17 13:41:57,753 WARN [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-3:null) (logid:d91c35e5) Expected 1 answers while executing SetMonitorServiceCommand but received 3
>>
>>
>>
>> 2022-10-17 13:41:57,938 INFO [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:null) (logid:1549c9a1) Fetching health check result for SEVERIPADDRESS and executing fresh checks: false
>>
>>
>>
>>
>>
>>
>>
>> On 10/17/22 14:29, Nicolas Vazquez wrote:
>>
>>
>>
>>
>>
>>
>>
>> Hi Groete,
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> Can you share the CloudStack agent logs from the KVM host?
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> Regards,
>>
>>
>>
>>
>>
>>
>>
>> Nicolas Vazquez
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> From: Granwille Strauss<gr...@namhost.com.INVALID> <gr...@namhost.com.INVALID> <gr...@namhost.com.INVALID> <gr...@namhost.com.INVALID> <gr...@namhost.com.INVALID>
>>
>>
>>
>>
>>
>>
>>
>> Date: Monday, 17 October 2022 at 08:56
>>
>>
>>
>>
>>
>>
>>
>> To:users@cloudstack.apache.org<ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org>
>>
>>
>>
>>
>>
>>
>>
>> Subject: Re: Recurring Snaps Failing
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> Hi
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> I ran a few more tests and can now finally make sense of the failure:
>>
>>
>>
>>
>>
>>
>>
>> 2022-10-17 13:39:08,556 DEBUG [o.a.c.s.d.d.CloudStackPrimaryDataStoreDriverImpl] (Work-Job-Executor-1:ctx-93cb161b job-806/job-807 ctx-3f9ccf45) (logid:8fd10b20) Failed to take snapshot: 23
>>
>>
>>
>>
>>
>>
>>
>> 2022-10-17 13:39:08,558 DEBUG [o.a.c.s.s.SnapshotServiceImpl] (Work-Job-Executor-1:ctx-93cb161b job-806/job-807 ctx-3f9ccf45) (logid:8fd10b20) create snapshot Test001 failed: com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:1, com.cloud.exception.OperationTimedoutException: Commands 7160441932542378205 to Host 1 timed out after 3600
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> Seems to be a time out error. But there's no firewalls on the KVM host and NFS server to prevent connections and it worked fine before and randomly just stopped.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> So I still have no idea where to solve this as all other necessary settings are in place. Any ideas?
>>
>>
>>
>>
>>
>>
>>
>> On 10/17/22 08:51, Granwille Strauss wrote:
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> Hi Guys
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> I enabled recurring snap shots for my Linux VMs and they all seem to be failing with the following error:
>>
>>
>>
>>
>>
>>
>>
>> root@hostname ~ $ grep error /var/log/cloudstack/management/management-server.log
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> 2022-10-17 03:01:29,474 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-27:ctx-99d7d702 job-796 ctx-f26a81fe) (logid:f9a61b00) Failed to create snapshot due to an internal error creating snapshot for volume 635fd5b5-bf14-4cf4-979c-be8a649d5189
>>
>>
>>
>>
>>
>>
>>
>> 2022-10-17 03:01:29,476 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-27:ctx-99d7d702 job-796) (logid:f9a61b00) Complete async job-796, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 635fd5b5-bf14-4cf4-979c-be8a649d5189"}
>>
>>
>>
>>
>>
>>
>>
>> 2022-10-17 03:01:29,478 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-26:ctx-c7865967 job-794 ctx-0700afb0) (logid:5541113c) Failed to create snapshot due to an internal error creating snapshot for volume 70ffef79-3aad-43ab-93d1-153fcd34ad15
>>
>>
>>
>>
>>
>>
>>
>> 2022-10-17 03:01:29,480 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-26:ctx-c7865967 job-794) (logid:5541113c) Complete async job-794, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 70ffef79-3aad-43ab-93d1-153fcd34ad15"}
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> All I am getting out of this is internal error and nothing else. I have checked other logs and I am not quite sure where else to look.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> Any assistance in this regard will be appreciated.
>>
>>
>>
>>
>>
>>
>>
>> --
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> Regards / Groete
>>
>>
>>
>>
>>
>>
>>
>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
>>
>>
>>
>>
>>
>>
>>
>> Granwille Strauss // Senior Systems Admin
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> e:granwille@namhost.com<ma...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com>
>>
>>
>>
>>
>>
>>
>>
>> m: +264 81 323 1260<tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260>
>>
>>
>>
>>
>>
>>
>>
>> w:www.namhost.com<http://www.namhost.com><http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com><http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com><http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>
>>
>>
>>
>>
>>
>>
>> Namhost Internet Services (Pty) Ltd,
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g><https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>
>>
>>
>>
>>
>>
>>
>> --
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> Regards / Groete
>>
>>
>>
>>
>>
>>
>>
>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
>>
>>
>>
>>
>>
>>
>>
>> Granwille Strauss // Senior Systems Admin
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> e:granwille@namhost.com<ma...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com>
>>
>>
>>
>>
>>
>>
>>
>> m: +264 81 323 1260<tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260>
>>
>>
>>
>>
>>
>>
>>
>> w:www.namhost.com<http://www.namhost.com><http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com><http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com><http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>
>>
>>
>>
>>
>>
>>
>> Namhost Internet Services (Pty) Ltd,
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g><https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> --
>>
>>
>>
>>
>>
>>
>>
>> Regards / Groete
>>
>>
>>
>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
>>
>>
>>
>> Granwille Strauss // Senior Systems Admin
>>
>>
>>
>>
>>
>>
>>
>> e:granwille@namhost.com<ma...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com>
>>
>>
>>
>> m: +264 81 323 1260<tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260>
>>
>>
>>
>> w:www.namhost.com<http://www.namhost.com><http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
>>
>>
>>
>>
>>
>>
>>
>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>
>>
>>
>>
>>
>>
>>
>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>
>>
>>
>> Namhost Internet Services (Pty) Ltd,
>>
>>
>>
>>
>>
>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g><https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>
>>
>>
>>
>>
>>
>>
>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>
>>
>>
>>
>>
>> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> --
>>
>> Regards / Groete
>>
>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
>>
>> Granwille Strauss // Senior Systems Admin
>>
>>
>>
>> e:granwille@namhost.com<ma...@namhost.com> <gr...@namhost.com>
>>
>> m: +264 81 323 1260<tel:+264813231260><tel:+264813231260>
>>
>> w:www.namhost.com<http://www.namhost.com><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
>>
>>
>>
>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>
>>
>>
>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>
>> Namhost Internet Services (Pty) Ltd,
>>
>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g><https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>
>>
>>
>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>
>>
>>
>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>
>> --
>>
>>
>>
>> Regards / Groete
>>
>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
>>
>> Granwille Strauss // Senior Systems Admin
>>
>>
>>
>> e:granwille@namhost.com<ma...@namhost.com> <gr...@namhost.com>
>>
>> m: +264 81 323 1260<tel:+264813231260><tel:+264813231260>
>>
>> w:www.namhost.com<http://www.namhost.com><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
>>
>>
>>
>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>
>>
>>
>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>
>> Namhost Internet Services (Pty) Ltd,
>>
>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g><https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>
>>
>>
>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and s
>>
>>
>>
>> --
>>
>> Regards / Groete
>>
>>
>>
>> <https://www.namhost.com><https://www.namhost.com> Granwille Strauss // Senior Systems Admin
>>
>>
>>
>> *e:*granwille@namhost.com<ma...@namhost.com>
>>
>> *m:* +264 81 323 1260 <+264813231260>
>>
>> *w:*www.namhost.com<http://www.namhost.com>
>>
>>
>>
>> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost>
>>
>> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/>
>>
>> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos>
>>
>> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>
>>
>>
>>
>>
>> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>
>>
>>
>> Namhost Internet Services (Pty) Ltd,
>>
>>
>>
>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g><https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>
>>
>>
>>
>>
>>
>>
>> The content of this message is confidential. If you have received it by
>>
>> mistake, please inform us by email reply and then delete the message. It is
>>
>> forbidden to copy, forward, or in any way reveal the contents of this
>>
>> message to anyone without our explicit consent. The integrity and security
>>
>> of this email cannot be guaranteed over the Internet. Therefore, the sender
>>
>> will not be held liable for any damage caused by the message. For our full
>>
>> privacy policy and disclaimers, please go tohttps://www.namhost.com/
>>
>> privacy-policy
>>
>>
>>
>> [image: Powered by AdSigner]
>>
>> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>
>>
>> --
>> Regards / Groete
>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
>> Granwille Strauss // Senior Systems Admin
>>
>> e:granwille@namhost.com<ma...@namhost.com>
>> m: +264 81 323 1260<tel:+264813231260>
>> w:www.namhost.com<https://www.namhost.com/>
>>
>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>
>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>> Namhost Internet Services (Pty) Ltd,
>>
>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>
>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>
>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>> --
>> Regards / Groete
>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
>> Granwille Strauss // Senior Systems Admin
>>
>> e:granwille@namhost.com<ma...@namhost.com>
>> m: +264 81 323 1260<tel:+264813231260>
>> w:www.namhost.com<https://www.namhost.com/>
>>
>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>
>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>> Namhost Internet Services (Pty) Ltd,
>>
>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>
>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>
>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>> --
>> Regards / Groete
>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
>> Granwille Strauss // Senior Systems Admin
>>
>> e:granwille@namhost.com<ma...@namhost.com>
>> m: +264 81 323 1260<tel:+264813231260>
>> w:www.namhost.com<https://www.namhost.com/>
>>
>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>
>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>> Namhost Internet Services (Pty) Ltd,
>>
>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>
>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>
>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>> --
>> Regards / Groete
>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
>> Granwille Strauss // Senior Systems Admin
>>
>> e:granwille@namhost.com<ma...@namhost.com>
>> m: +264 81 323 1260<tel:+264813231260>
>> w:www.namhost.com<https://www.namhost.com/>
>>
>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>
>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>> Namhost Internet Services (Pty) Ltd,
>>
>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>
>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>
>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>> --
>> Regards / Groete
>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
>> Granwille Strauss // Senior Systems Admin
>>
>> e:granwille@namhost.com<ma...@namhost.com>
>> m: +264 81 323 1260<tel:+264813231260>
>> w:www.namhost.com<https://www.namhost.com/>
>>
>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>
>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>> Namhost Internet Services (Pty) Ltd,
>>
>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>
>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>
>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>> --
>> Regards / Groete
>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
>> Granwille Strauss // Senior Systems Admin
>>
>> e:granwille@namhost.com<ma...@namhost.com>
>> m: +264 81 323 1260<tel:+264813231260>
>> w:www.namhost.com<https://www.namhost.com/>
>>
>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>
>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>> Namhost Internet Services (Pty) Ltd,
>>
>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>
>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>
>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>> --
>>
>> Regards / Groete
>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
>> Granwille Strauss // Senior Systems Admin
>>
>> e:granwille@namhost.com<ma...@namhost.com>
>> m: +264 81 323 1260<tel:+264813231260>
>> w:www.namhost.com<https://www.namhost.com/>
>>
>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>
>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>> Namhost Internet Services (Pty) Ltd,
>>
>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>
>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>
>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>> --
>>
>> Regards / Groete
>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
>> Granwille Strauss // Senior Systems Admin
>>
>> e:granwille@namhost.com<ma...@namhost.com>
>> m: +264 81 323 1260<tel:+264813231260>
>> w:www.namhost.com<https://www.namhost.com/>
>>
>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>
>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>> Namhost Internet Services (Pty) Ltd,
>>
>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>
>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>
>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>
> --
> Regards / Groete
>
> <https://www.namhost.com> Granwille Strauss // Senior Systems Admin
>
> *e:* granwille@namhost.com
> *m:* +264 81 323 1260 <tel:+264813231260>
> *w:* www.namhost.com <https://www.namhost.com/>
>
> <https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>
> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>
> Namhost Internet Services (Pty) Ltd,
> 24 Black Eagle Rd, Hermanus, 7210, RSA
>
>
> The content of this message is confidential. If you have received it
> by mistake, please inform us by email reply and then delete the
> message. It is forbidden to copy, forward, or in any way reveal the
> contents of this message to anyone without our explicit consent. The
> integrity and security of this email cannot be guaranteed over the
> Internet. Therefore, the sender will not be held liable for any damage
> caused by the message. For our full privacy policy and disclaimers,
> please go to https://www.namhost.com/privacy-policy
>
> Powered by AdSigner
> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
--
Regards / Groete
<https://www.namhost.com> Granwille Strauss // Senior Systems Admin
*e:* granwille@namhost.com
*m:* +264 81 323 1260 <tel:+264813231260>
*w:* www.namhost.com <https://www.namhost.com/>
<https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
Namhost Internet Services (Pty) Ltd,
24 Black Eagle Rd, Hermanus, 7210, RSA
The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It
is forbidden to copy, forward, or in any way reveal the contents of this
message to anyone without our explicit consent. The integrity and
security of this email cannot be guaranteed over the Internet.
Therefore, the sender will not be held liable for any damage caused by
the message. For our full privacy policy and disclaimers, please go to
https://www.namhost.com/privacy-policy
Powered by AdSigner
<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
Re: Recurring Snaps Failing
Posted by Granwille Strauss <gr...@namhost.com.INVALID>.
Hi
Alright, lets give that a shot. My `wait` in global config was set as
1800 seconds. I have increased it to 3600 seconds, restarted
cloudstack-management and started a volume snapshot. Lets see. Attached
is a screenshot of all wait settings I have in place, please let me know
if I missed anything?
On 10/18/22 15:31, Nicolas Vazquez wrote:
> Hi,
>
> As a final test I would suggest increasing the value of the ‘wait’ configuration. Checking the logs, it does not ever reach the point in which the snapshot is copied to the snapshots directory on primary storage and it may be taking too long. Currently, the operation is being timeout after 3600 seconds (2 * wait value?), can you try increasing it? It would require restarting the management server after applying the new value. If this doesn’t help please create a new issue on Github
>
> Regards,
> Nicolas Vazquez
>
>
> From: Granwille Strauss<gr...@namhost.com>
> Date: Tuesday, 18 October 2022 at 09:31
> To: Wei ZHOU<us...@gmail.com>
> Cc: Nicolas Vazquez<Ni...@shapeblue.com>,users@cloudstack.apache.org <us...@cloudstack.apache.org>
> Subject: Re: Recurring Snaps Failing
>
> Hi Guys
>
> Was there anything else I could do to get this fixed? Should I report this on Github? Also, is it possible to make recurring VM Snapshots and not only Volume Snapshots?
> On 10/18/22 11:43, Granwille Strauss wrote:
>
> Hi Wei
>
> Correct, yes VM Snapshot is 100% successful, its only volume snapshot that fails and its at the conversion it seems. As requested:
> root@HOSTNAME ~ $ virsh domblklist i-2-19-VM
> Target Source
> ------------------------------------------------------------------------------------------
> vda /mnt/81ffca3a-9775-375d-a1c0-9504c0ec3d89/635fd5b5-bf14-4cf4-979c-be8a649d5189
> hdc -
>
> root@HOSTNAME ~ $ qemu-img info /mnt/81ffca3a-9775-375d-a1c0-9504c0ec3d89/635fd5b5-bf14-4cf4-979c-be8a649d5189
> qemu-img: Could not open '/mnt/81ffca3a-9775-375d-a1c0-9504c0ec3d89/635fd5b5-bf14-4cf4-979c-be8a649d5189': Failed to get shared "write" lock
> Is another process using the image [/mnt/81ffca3a-9775-375d-a1c0-9504c0ec3d89/635fd5b5-bf14-4cf4-979c-be8a649d5189]?
> root@athena ~ $
>
> Seems I cannot get you the details, however would this help?
> root@HOSTNAME /mnt/81ffca3a-9775-375d-a1c0-9504c0ec3d89 $ du * -sh
> 48G 340cc676-4683-4424-beb7-8b33a5cb3dfe
> 42G 4a56546f-4f63-41c2-a314-27619fe5c2dc
> 15G 635fd5b5-bf14-4cf4-979c-be8a649d5189
> 21G 70ffef79-3aad-43ab-93d1-153fcd34ad15
> 26G 777b5149-50c0-4c0d-944b-873050c85419
> 20G 867a61e8-4635-40e4-b1b1-c94b2a325b49
> 4.8G b21551c6-8276-4e58-841e-4e5a02c0e163
> 11G d2c4feca-da5c-44b3-8ecc-40a01fd6c243
> 40G f50c0bca-3ae2-48ee-aa9d-0e2d4ec6ba92
> 24G f9336d09-0b3d-42d8-9a24-883b022b6712
> 8.9G f962cb2c-e03b-4ed2-a035-1870a367aa46
> 47G fd573618-885e-48c7-86e6-b019e2311d76
> 8.0K KVMHA
> 15G snapshots
> root@HOSTNAME /mnt/81ffca3a-9775-375d-a1c0-9504c0ec3d89 $
>
>
> I have also added latest agent.log from today.
> On 10/18/22 11:07, Wei ZHOU wrote:
> Hi,
>
> You can get the path of qcow2 image by "virsh domblklist i-2-19-VM". The QCOW2 image does not have .qcow2 as a suffix.
>
> From what I see, a vm snapshot has been created 2022-10-18 09:44:21 +0200, which is good as taking the snapshot started at the same time.
> so timed out when converting the vm snapshot to a qcow2 image (ons secondary storage) by "qemu-img convert".
>
> ps: the agent.log you attached is out-of-date. Please share the agent.log in few minutes after "2022-10-18 09:44:21"
>
>
> -Wei
>
>
>
>
> On Tue, 18 Oct 2022 at 10:53, Granwille Strauss <gr...@namhost.com>> wrote:
>
> Hi Wei
>
> Thank you for the update. Thus far all the snapshot tests I took were volume snaps and majority were when they are online and end up failing. However, there's one VM where it seems to work just fine, whether its online of offline and I am not sure what's different from this VM as all other VM use the same Almalinux template. Its just this one VM where it works as expected.
>
> In any case, I did a VM snapshot now with memory on my Ubuntu vm while it was online and the VM Snap was successful, no issues there. I ran a volume snap right after on the same ubuntu vm and it failed again as before, see attached screenshot. So VM snapshots seem to work fine, this tells me the issue is most likely when its converted to QCOW2.
>
> In any case, please see answers to your questions below:
>
> 1. I am not finding any .qcow2 images on the KVM server to list the QCOW2 size, or am I missing something?
> root@HOSTNAME ~ $ find / -type f -iname '*.qcow2'
> root@HOSTNAME ~ $
> As requested, here's a snapshot list of the current vm I am testing on. Please let me know if you needed something specific?
> root@HOSTNAME ~ $ virsh snapshot-list i-2-19-VM
> Name Creation Time State
> -----------------------------------------------------------------------------------
> 0843e6b1-cb12-495c-8d5f-35c9a3338db7 2022-10-17 18:41:08 +0200 disk-snapshot
> 2e77acb7-1be4-4417-858e-a1d0890f44f0 2022-10-17 19:57:03 +0200 disk-snapshot
> 68476e2a-aec8-4637-9c4b-2dc49d79bd82 2022-10-18 09:44:21 +0200 disk-snapshot
> c110058b-4661-42c0-b648-a54f4ba4beb7 2022-10-17 22:34:21 +0200 disk-snapshot
> i-2-19-VM_VS_20221018072924 2022-10-18 09:29:25 +0200 running
>
> 2. Attached is a txt file from yesterdays agent.log. This is all entries that happened between CopyCommand and CopyCmdAnswer. There are zero errors, as you will see.
>
> 3. The 'backup.snapshot.wait' setting is currently at 21600 seconds = 6 hours, should this still be increased? Also, not sure if 'copy.volume.wait' is relevant, but this is currently at 10800 seconds.
> On 10/18/22 09:30, Wei ZHOU wrote:
> Hi Granwille,
>
> The process of volume snapshot on running vms and stop vms are different. For running vms, cloudstack will first take a vm snapshot and convert the vm snapshot into a qcow2 image.
> It is normal that snapshotting a running vm takes longer time than a stopped vm, because
> (1) there is an extra step to take a vm snapshot. Depending on the memory size, the step might take a few minutes.
> (3) vm snapshot is saved in QCOW2 image (it is called Internal snapshots). When taking a vm snapshot, the QCOW2 image will enlarge. The image will not shrink when the vm snapshot is removed.
> If you have taken vm snapshot (or volume snapshot on running vms) multiple times, The QCOW2 image might be very big. The whole process (including vm snapshotting and qemu-img convert) might exceed the timeout limit (3600 seconds = 1 hour).
>
> can you please
> (1) check the QCOW2 image size and list the snapshot of QCOW2 image (qemu-img info), list vm snapshot of the vm (virsh snapshot-list)
> (2) check the logs between CopyCommand and CopyCmdAnswer in agent.log
> (3) increase backup.snapshot.wait in global configuration and retry ("wait" in "CopyCommand" should be changed accordingly) ?
>
>
> -Wei
>
>
>
> On Tue, 18 Oct 2022 at 00:39, Granwille Strauss <gr...@namhost.com>> wrote:
>
> Hi Guys
>
> I tested another Linux VM Almalinux VM, turned off the VM and made a successful snap. I turned it back online again and run another successful snap again. Now I run a snap on my Ubuntu VM and it failed. So just to be sure it wasn't just that one specific Almalinux VM that was working, I ran another snap on another Almalinux VM while it was online and too also failed. So I am not sure why recurring snaps is only working for this one VM and not the rest.
>
> Should I log this on GITHUB?
> On 10/17/22 21:55, Granwille Strauss wrote:
>
> Sorry, seems my attachments were too big, you can download via Wetransfer:https://we.tl/t-9hUekbj86S
>
> And yes, I have the kvm.snapshot.enabled enabled in global settings too, for live snapping.
> On 10/17/22 21:51, Granwille Strauss wrote:
>
> So it seems when VM is powered off, snap is successful. See attached logs for the successful snap, when VM is offline.
> On 10/17/22 21:38, Granwille Strauss wrote:
>
> Here's the last 10 minutes before the error is triggered in the logs, this was the test run before switching off VM. That test is still running and will provide its logs shortly, however, I am confident, it will show same results.
> On 10/17/22 21:22, Granwille Strauss wrote:
>
> Hi Wei
>
> Please see attached. Please keep in mind, this worked 100% a few days ago when I tested on VM recurring snaps and it was perfect then. So I enabled recurring snaps for all 10 linux VMs I host. And I set them all to daily at 2AM.
> They failed after the first day. So I figured maybe too many concurrent jobs running at the same time.
>
> So I split them all to once weekly at different days at different hours. And still it failed. And now when doing it manually it fails too. But for some reason the snap ends up in /primary location is just returns an error for some reason.
> On 10/17/22 21:13, Wei ZHOU wrote:
>
> Hi,
>
>
>
> What is the kvm host os and version ?
>
>
>
> -Wei
>
>
>
> On Monday, 17 October 2022, Granwille Strauss<gr...@namhost.com.invalid>
>
> wrote:
>
>
>
> The error seems to happen exactly at 1 hour intervals. All of them have
>
> failed exactly at one hour. Also, these are all volume snapshots that I am
>
> taking, so the logs I sent you so far are all in this regard. They are not
>
> VM snaphots, nor are they powered off. I am running a new volume snap shot
>
> again, this time VM powered off lets see.
>
>
>
> I have also rebooted my entire KVM server and my management server. So far
>
> zero luck.
>
> On 10/17/22 20:26, Nicolas Vazquez wrote:
>
>
>
> Can you try taking a manual volume snapshot and share both mgmt and agent logs please?
>
>
>
> Regards,
>
> Nicolas Vazquez
>
>
>
>
>
> From: Granwille Strauss<gr...@namhost.com> <gr...@namhost.com>
>
> Date: Monday, 17 October 2022 at 13:15
>
> To:users@cloudstack.apache.org<ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org>
>
> Cc: Nicolas Vazquez<Ni...@shapeblue.com> <Ni...@shapeblue.com>
>
> Subject: Re: Recurring Snaps Failing
>
>
>
> Hi
>
>
>
> I also see another issue with snaps, I deleted all the failed snaps via the cloud management UI. But to my surprise, they're all still at /mnt/primary/snapshots/ see attached screenshot. Something odd is going on with snapshots.
>
> On 10/17/22 17:35, Granwille Strauss wrote:
>
>
>
> Hi
>
>
>
> I am not 100% if I enabled debug correctly, I followed thishttps://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+agent+debug and ran
>
> sed -i 's/INFO/DEBUG/g' /etc/cloudstack/agent/log4j-cloud.xml
>
>
>
> Attached are the two log files. Same error. I reboot SSVM & vRouter prior no luck.
>
> On 10/17/22 16:00, Granwille Strauss wrote:
>
>
>
> Hi
>
>
>
> No KVM is on its own server node and management on a different server node. However, management is is primary and secondary storage via NFS. I will enable DEBUG and re-run manual snap to replicate the error.
>
> On 10/17/22 15:42, Nicolas Vazquez wrote:
>
> Thanks, are the KVM agent and management server running on the same host? Can you increase the log level on the KVM agent logs to DEBUG and retry?
>
>
>
> Regards,
>
> Nicolas Vazquez
>
>
>
>
>
>
>
>
>
>
>
>
>
> From: Granwille Strauss<gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com>
>
> Date: Monday, 17 October 2022 at 10:04
>
> To:users@cloudstack.apache.org<ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org>
>
> Cc: Nicolas Vazquez<Ni...@shapeblue.com> <Ni...@shapeblue.com> <Ni...@shapeblue.com>
>
> Subject: Re: Recurring Snaps Failing
>
>
>
> Please find attached TXT
>
> On 10/17/22 14:54, Nicolas Vazquez wrote:
>
>
>
> Ok, seems like the command is not reaching the KVM host, can you share the full management server logs for job-796?
>
>
>
>
>
>
>
> Regards,
>
>
>
> Nicolas Vazquez
>
>
>
>
>
>
>
>
>
>
>
> From: Granwille Strauss<gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com>
>
>
>
> Date: Monday, 17 October 2022 at 09:32
>
>
>
> To:users@cloudstack.apache.org<ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org>
>
>
>
> Cc: Nicolas Vazquez<Ni...@shapeblue.com> <Ni...@shapeblue.com> <Ni...@shapeblue.com>
>
>
>
> Subject: Re: Recurring Snaps Failing
>
>
>
>
>
>
>
> Hi
>
>
>
>
>
>
>
> This is all agent threw during the snap process nothing else:
>
>
>
> 2022-10-17 13:39:12,375 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:a864f9f6) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
>
>
>
> 2022-10-17 13:40:13,276 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:445216f3) Trying to fetch storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297 from libvirt
>
>
>
> 2022-10-17 13:40:13,289 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:445216f3) Asking libvirt to refresh storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297
>
>
>
> 2022-10-17 13:40:13,378 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:445216f3) Trying to fetch storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89 from libvirt
>
>
>
> 2022-10-17 13:40:13,393 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:445216f3) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
>
>
>
> 2022-10-17 13:41:13,950 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:87ecfee0) Trying to fetch storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297 from libvirt
>
>
>
> 2022-10-17 13:41:13,964 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:87ecfee0) Asking libvirt to refresh storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297
>
>
>
> 2022-10-17 13:41:14,054 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:87ecfee0) Trying to fetch storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89 from libvirt
>
>
>
> 2022-10-17 13:41:14,067 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:87ecfee0) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
>
>
>
> 2022-10-17 13:41:57,753 WARN [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-3:null) (logid:d91c35e5) Expected 1 answers while executing SetMonitorServiceCommand but received 3
>
>
>
> 2022-10-17 13:41:57,938 INFO [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:null) (logid:1549c9a1) Fetching health check result for SEVERIPADDRESS and executing fresh checks: false
>
>
>
>
>
>
>
> On 10/17/22 14:29, Nicolas Vazquez wrote:
>
>
>
>
>
>
>
> Hi Groete,
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> Can you share the CloudStack agent logs from the KVM host?
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> Regards,
>
>
>
>
>
>
>
> Nicolas Vazquez
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> From: Granwille Strauss<gr...@namhost.com.INVALID> <gr...@namhost.com.INVALID> <gr...@namhost.com.INVALID> <gr...@namhost.com.INVALID> <gr...@namhost.com.INVALID>
>
>
>
>
>
>
>
> Date: Monday, 17 October 2022 at 08:56
>
>
>
>
>
>
>
> To:users@cloudstack.apache.org<ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org>
>
>
>
>
>
>
>
> Subject: Re: Recurring Snaps Failing
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> Hi
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> I ran a few more tests and can now finally make sense of the failure:
>
>
>
>
>
>
>
> 2022-10-17 13:39:08,556 DEBUG [o.a.c.s.d.d.CloudStackPrimaryDataStoreDriverImpl] (Work-Job-Executor-1:ctx-93cb161b job-806/job-807 ctx-3f9ccf45) (logid:8fd10b20) Failed to take snapshot: 23
>
>
>
>
>
>
>
> 2022-10-17 13:39:08,558 DEBUG [o.a.c.s.s.SnapshotServiceImpl] (Work-Job-Executor-1:ctx-93cb161b job-806/job-807 ctx-3f9ccf45) (logid:8fd10b20) create snapshot Test001 failed: com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:1, com.cloud.exception.OperationTimedoutException: Commands 7160441932542378205 to Host 1 timed out after 3600
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> Seems to be a time out error. But there's no firewalls on the KVM host and NFS server to prevent connections and it worked fine before and randomly just stopped.
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> So I still have no idea where to solve this as all other necessary settings are in place. Any ideas?
>
>
>
>
>
>
>
> On 10/17/22 08:51, Granwille Strauss wrote:
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> Hi Guys
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> I enabled recurring snap shots for my Linux VMs and they all seem to be failing with the following error:
>
>
>
>
>
>
>
> root@hostname ~ $ grep error /var/log/cloudstack/management/management-server.log
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> 2022-10-17 03:01:29,474 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-27:ctx-99d7d702 job-796 ctx-f26a81fe) (logid:f9a61b00) Failed to create snapshot due to an internal error creating snapshot for volume 635fd5b5-bf14-4cf4-979c-be8a649d5189
>
>
>
>
>
>
>
> 2022-10-17 03:01:29,476 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-27:ctx-99d7d702 job-796) (logid:f9a61b00) Complete async job-796, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 635fd5b5-bf14-4cf4-979c-be8a649d5189"}
>
>
>
>
>
>
>
> 2022-10-17 03:01:29,478 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-26:ctx-c7865967 job-794 ctx-0700afb0) (logid:5541113c) Failed to create snapshot due to an internal error creating snapshot for volume 70ffef79-3aad-43ab-93d1-153fcd34ad15
>
>
>
>
>
>
>
> 2022-10-17 03:01:29,480 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-26:ctx-c7865967 job-794) (logid:5541113c) Complete async job-794, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 70ffef79-3aad-43ab-93d1-153fcd34ad15"}
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> All I am getting out of this is internal error and nothing else. I have checked other logs and I am not quite sure where else to look.
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> Any assistance in this regard will be appreciated.
>
>
>
>
>
>
>
> --
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> Regards / Groete
>
>
>
>
>
>
>
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
>
>
>
>
>
>
>
> Granwille Strauss // Senior Systems Admin
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> e:granwille@namhost.com<ma...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com>
>
>
>
>
>
>
>
> m: +264 81 323 1260<tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260>
>
>
>
>
>
>
>
> w:www.namhost.com<http://www.namhost.com><http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com><http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com><http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>
>
>
>
>
>
>
> Namhost Internet Services (Pty) Ltd,
>
>
>
>
>
>
>
>
>
>
>
>
>
> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g><https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>
>
>
>
>
>
>
> --
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> Regards / Groete
>
>
>
>
>
>
>
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
>
>
>
>
>
>
>
> Granwille Strauss // Senior Systems Admin
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> e:granwille@namhost.com<ma...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com>
>
>
>
>
>
>
>
> m: +264 81 323 1260<tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260>
>
>
>
>
>
>
>
> w:www.namhost.com<http://www.namhost.com><http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com><http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com><http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>
>
>
>
>
>
>
> Namhost Internet Services (Pty) Ltd,
>
>
>
>
>
>
>
>
>
>
>
>
>
> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g><https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> --
>
>
>
>
>
>
>
> Regards / Groete
>
>
>
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
>
>
>
> Granwille Strauss // Senior Systems Admin
>
>
>
>
>
>
>
> e:granwille@namhost.com<ma...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com>
>
>
>
> m: +264 81 323 1260<tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260>
>
>
>
> w:www.namhost.com<http://www.namhost.com><http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
>
>
>
>
>
>
>
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>
>
>
>
>
>
>
> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>
>
>
> Namhost Internet Services (Pty) Ltd,
>
>
>
>
>
> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g><https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>
>
>
>
>
>
>
> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>
>
>
>
>
> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> --
>
> Regards / Groete
>
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
>
> Granwille Strauss // Senior Systems Admin
>
>
>
> e:granwille@namhost.com<ma...@namhost.com> <gr...@namhost.com>
>
> m: +264 81 323 1260<tel:+264813231260><tel:+264813231260>
>
> w:www.namhost.com<http://www.namhost.com><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
>
>
>
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>
>
>
> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>
> Namhost Internet Services (Pty) Ltd,
>
> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g><https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>
>
>
> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>
>
>
> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>
> --
>
>
>
> Regards / Groete
>
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
>
> Granwille Strauss // Senior Systems Admin
>
>
>
> e:granwille@namhost.com<ma...@namhost.com> <gr...@namhost.com>
>
> m: +264 81 323 1260<tel:+264813231260><tel:+264813231260>
>
> w:www.namhost.com<http://www.namhost.com><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
>
>
>
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>
>
>
> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>
> Namhost Internet Services (Pty) Ltd,
>
> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g><https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>
>
>
> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and s
>
>
>
> --
>
> Regards / Groete
>
>
>
> <https://www.namhost.com><https://www.namhost.com> Granwille Strauss // Senior Systems Admin
>
>
>
> *e:*granwille@namhost.com<ma...@namhost.com>
>
> *m:* +264 81 323 1260 <+264813231260>
>
> *w:*www.namhost.com<http://www.namhost.com>
>
>
>
> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost>
>
> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/>
>
> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos>
>
> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>
>
>
>
>
> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>
>
>
> Namhost Internet Services (Pty) Ltd,
>
>
>
> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g><https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>
>
>
>
>
>
>
> The content of this message is confidential. If you have received it by
>
> mistake, please inform us by email reply and then delete the message. It is
>
> forbidden to copy, forward, or in any way reveal the contents of this
>
> message to anyone without our explicit consent. The integrity and security
>
> of this email cannot be guaranteed over the Internet. Therefore, the sender
>
> will not be held liable for any damage caused by the message. For our full
>
> privacy policy and disclaimers, please go tohttps://www.namhost.com/
>
> privacy-policy
>
>
>
> [image: Powered by AdSigner]
>
> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>
>
> --
> Regards / Groete
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
> Granwille Strauss // Senior Systems Admin
>
> e:granwille@namhost.com<ma...@namhost.com>
> m: +264 81 323 1260<tel:+264813231260>
> w:www.namhost.com<https://www.namhost.com/>
>
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>
> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
> Namhost Internet Services (Pty) Ltd,
>
> 24 Black Eagle Rd, Hermanus, 7210, RSA
>
> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>
> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
> --
> Regards / Groete
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
> Granwille Strauss // Senior Systems Admin
>
> e:granwille@namhost.com<ma...@namhost.com>
> m: +264 81 323 1260<tel:+264813231260>
> w:www.namhost.com<https://www.namhost.com/>
>
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>
> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
> Namhost Internet Services (Pty) Ltd,
>
> 24 Black Eagle Rd, Hermanus, 7210, RSA
>
> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>
> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
> --
> Regards / Groete
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
> Granwille Strauss // Senior Systems Admin
>
> e:granwille@namhost.com<ma...@namhost.com>
> m: +264 81 323 1260<tel:+264813231260>
> w:www.namhost.com<https://www.namhost.com/>
>
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>
> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
> Namhost Internet Services (Pty) Ltd,
>
> 24 Black Eagle Rd, Hermanus, 7210, RSA
>
> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>
> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
> --
> Regards / Groete
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
> Granwille Strauss // Senior Systems Admin
>
> e:granwille@namhost.com<ma...@namhost.com>
> m: +264 81 323 1260<tel:+264813231260>
> w:www.namhost.com<https://www.namhost.com/>
>
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>
> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
> Namhost Internet Services (Pty) Ltd,
>
> 24 Black Eagle Rd, Hermanus, 7210, RSA
>
> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>
> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
> --
> Regards / Groete
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
> Granwille Strauss // Senior Systems Admin
>
> e:granwille@namhost.com<ma...@namhost.com>
> m: +264 81 323 1260<tel:+264813231260>
> w:www.namhost.com<https://www.namhost.com/>
>
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>
> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
> Namhost Internet Services (Pty) Ltd,
>
> 24 Black Eagle Rd, Hermanus, 7210, RSA
>
> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>
> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
> --
> Regards / Groete
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
> Granwille Strauss // Senior Systems Admin
>
> e:granwille@namhost.com<ma...@namhost.com>
> m: +264 81 323 1260<tel:+264813231260>
> w:www.namhost.com<https://www.namhost.com/>
>
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>
> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
> Namhost Internet Services (Pty) Ltd,
>
> 24 Black Eagle Rd, Hermanus, 7210, RSA
>
> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>
> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
> --
>
> Regards / Groete
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
> Granwille Strauss // Senior Systems Admin
>
> e:granwille@namhost.com<ma...@namhost.com>
> m: +264 81 323 1260<tel:+264813231260>
> w:www.namhost.com<https://www.namhost.com/>
>
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>
> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
> Namhost Internet Services (Pty) Ltd,
>
> 24 Black Eagle Rd, Hermanus, 7210, RSA
>
> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>
> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
> --
>
> Regards / Groete
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
> Granwille Strauss // Senior Systems Admin
>
> e:granwille@namhost.com<ma...@namhost.com>
> m: +264 81 323 1260<tel:+264813231260>
> w:www.namhost.com<https://www.namhost.com/>
>
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>
> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
> Namhost Internet Services (Pty) Ltd,
>
> 24 Black Eagle Rd, Hermanus, 7210, RSA
>
> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>
> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>
--
Regards / Groete
<https://www.namhost.com> Granwille Strauss // Senior Systems Admin
*e:* granwille@namhost.com
*m:* +264 81 323 1260 <tel:+264813231260>
*w:* www.namhost.com <https://www.namhost.com/>
<https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
Namhost Internet Services (Pty) Ltd,
24 Black Eagle Rd, Hermanus, 7210, RSA
The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It
is forbidden to copy, forward, or in any way reveal the contents of this
message to anyone without our explicit consent. The integrity and
security of this email cannot be guaranteed over the Internet.
Therefore, the sender will not be held liable for any damage caused by
the message. For our full privacy policy and disclaimers, please go to
https://www.namhost.com/privacy-policy
Powered by AdSigner
<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
Re: Recurring Snaps Failing
Posted by Nicolas Vazquez <Ni...@shapeblue.com>.
Hi,
As a final test I would suggest increasing the value of the ‘wait’ configuration. Checking the logs, it does not ever reach the point in which the snapshot is copied to the snapshots directory on primary storage and it may be taking too long. Currently, the operation is being timeout after 3600 seconds (2 * wait value?), can you try increasing it? It would require restarting the management server after applying the new value. If this doesn’t help please create a new issue on Github
Regards,
Nicolas Vazquez
From: Granwille Strauss <gr...@namhost.com>
Date: Tuesday, 18 October 2022 at 09:31
To: Wei ZHOU <us...@gmail.com>
Cc: Nicolas Vazquez <Ni...@shapeblue.com>, users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Re: Recurring Snaps Failing
Hi Guys
Was there anything else I could do to get this fixed? Should I report this on Github? Also, is it possible to make recurring VM Snapshots and not only Volume Snapshots?
On 10/18/22 11:43, Granwille Strauss wrote:
Hi Wei
Correct, yes VM Snapshot is 100% successful, its only volume snapshot that fails and its at the conversion it seems. As requested:
root@HOSTNAME ~ $ virsh domblklist i-2-19-VM
Target Source
------------------------------------------------------------------------------------------
vda /mnt/81ffca3a-9775-375d-a1c0-9504c0ec3d89/635fd5b5-bf14-4cf4-979c-be8a649d5189
hdc -
root@HOSTNAME ~ $ qemu-img info /mnt/81ffca3a-9775-375d-a1c0-9504c0ec3d89/635fd5b5-bf14-4cf4-979c-be8a649d5189
qemu-img: Could not open '/mnt/81ffca3a-9775-375d-a1c0-9504c0ec3d89/635fd5b5-bf14-4cf4-979c-be8a649d5189': Failed to get shared "write" lock
Is another process using the image [/mnt/81ffca3a-9775-375d-a1c0-9504c0ec3d89/635fd5b5-bf14-4cf4-979c-be8a649d5189]?
root@athena ~ $
Seems I cannot get you the details, however would this help?
root@HOSTNAME /mnt/81ffca3a-9775-375d-a1c0-9504c0ec3d89 $ du * -sh
48G 340cc676-4683-4424-beb7-8b33a5cb3dfe
42G 4a56546f-4f63-41c2-a314-27619fe5c2dc
15G 635fd5b5-bf14-4cf4-979c-be8a649d5189
21G 70ffef79-3aad-43ab-93d1-153fcd34ad15
26G 777b5149-50c0-4c0d-944b-873050c85419
20G 867a61e8-4635-40e4-b1b1-c94b2a325b49
4.8G b21551c6-8276-4e58-841e-4e5a02c0e163
11G d2c4feca-da5c-44b3-8ecc-40a01fd6c243
40G f50c0bca-3ae2-48ee-aa9d-0e2d4ec6ba92
24G f9336d09-0b3d-42d8-9a24-883b022b6712
8.9G f962cb2c-e03b-4ed2-a035-1870a367aa46
47G fd573618-885e-48c7-86e6-b019e2311d76
8.0K KVMHA
15G snapshots
root@HOSTNAME /mnt/81ffca3a-9775-375d-a1c0-9504c0ec3d89 $
I have also added latest agent.log from today.
On 10/18/22 11:07, Wei ZHOU wrote:
Hi,
You can get the path of qcow2 image by "virsh domblklist i-2-19-VM". The QCOW2 image does not have .qcow2 as a suffix.
From what I see, a vm snapshot has been created 2022-10-18 09:44:21 +0200, which is good as taking the snapshot started at the same time.
so timed out when converting the vm snapshot to a qcow2 image (ons secondary storage) by "qemu-img convert".
ps: the agent.log you attached is out-of-date. Please share the agent.log in few minutes after "2022-10-18 09:44:21"
-Wei
On Tue, 18 Oct 2022 at 10:53, Granwille Strauss <gr...@namhost.com>> wrote:
Hi Wei
Thank you for the update. Thus far all the snapshot tests I took were volume snaps and majority were when they are online and end up failing. However, there's one VM where it seems to work just fine, whether its online of offline and I am not sure what's different from this VM as all other VM use the same Almalinux template. Its just this one VM where it works as expected.
In any case, I did a VM snapshot now with memory on my Ubuntu vm while it was online and the VM Snap was successful, no issues there. I ran a volume snap right after on the same ubuntu vm and it failed again as before, see attached screenshot. So VM snapshots seem to work fine, this tells me the issue is most likely when its converted to QCOW2.
In any case, please see answers to your questions below:
1. I am not finding any .qcow2 images on the KVM server to list the QCOW2 size, or am I missing something?
root@HOSTNAME ~ $ find / -type f -iname '*.qcow2'
root@HOSTNAME ~ $
As requested, here's a snapshot list of the current vm I am testing on. Please let me know if you needed something specific?
root@HOSTNAME ~ $ virsh snapshot-list i-2-19-VM
Name Creation Time State
-----------------------------------------------------------------------------------
0843e6b1-cb12-495c-8d5f-35c9a3338db7 2022-10-17 18:41:08 +0200 disk-snapshot
2e77acb7-1be4-4417-858e-a1d0890f44f0 2022-10-17 19:57:03 +0200 disk-snapshot
68476e2a-aec8-4637-9c4b-2dc49d79bd82 2022-10-18 09:44:21 +0200 disk-snapshot
c110058b-4661-42c0-b648-a54f4ba4beb7 2022-10-17 22:34:21 +0200 disk-snapshot
i-2-19-VM_VS_20221018072924 2022-10-18 09:29:25 +0200 running
2. Attached is a txt file from yesterdays agent.log. This is all entries that happened between CopyCommand and CopyCmdAnswer. There are zero errors, as you will see.
3. The 'backup.snapshot.wait' setting is currently at 21600 seconds = 6 hours, should this still be increased? Also, not sure if 'copy.volume.wait' is relevant, but this is currently at 10800 seconds.
On 10/18/22 09:30, Wei ZHOU wrote:
Hi Granwille,
The process of volume snapshot on running vms and stop vms are different. For running vms, cloudstack will first take a vm snapshot and convert the vm snapshot into a qcow2 image.
It is normal that snapshotting a running vm takes longer time than a stopped vm, because
(1) there is an extra step to take a vm snapshot. Depending on the memory size, the step might take a few minutes.
(3) vm snapshot is saved in QCOW2 image (it is called Internal snapshots). When taking a vm snapshot, the QCOW2 image will enlarge. The image will not shrink when the vm snapshot is removed.
If you have taken vm snapshot (or volume snapshot on running vms) multiple times, The QCOW2 image might be very big. The whole process (including vm snapshotting and qemu-img convert) might exceed the timeout limit (3600 seconds = 1 hour).
can you please
(1) check the QCOW2 image size and list the snapshot of QCOW2 image (qemu-img info), list vm snapshot of the vm (virsh snapshot-list)
(2) check the logs between CopyCommand and CopyCmdAnswer in agent.log
(3) increase backup.snapshot.wait in global configuration and retry ("wait" in "CopyCommand" should be changed accordingly) ?
-Wei
On Tue, 18 Oct 2022 at 00:39, Granwille Strauss <gr...@namhost.com>> wrote:
Hi Guys
I tested another Linux VM Almalinux VM, turned off the VM and made a successful snap. I turned it back online again and run another successful snap again. Now I run a snap on my Ubuntu VM and it failed. So just to be sure it wasn't just that one specific Almalinux VM that was working, I ran another snap on another Almalinux VM while it was online and too also failed. So I am not sure why recurring snaps is only working for this one VM and not the rest.
Should I log this on GITHUB?
On 10/17/22 21:55, Granwille Strauss wrote:
Sorry, seems my attachments were too big, you can download via Wetransfer: https://we.tl/t-9hUekbj86S
And yes, I have the kvm.snapshot.enabled enabled in global settings too, for live snapping.
On 10/17/22 21:51, Granwille Strauss wrote:
So it seems when VM is powered off, snap is successful. See attached logs for the successful snap, when VM is offline.
On 10/17/22 21:38, Granwille Strauss wrote:
Here's the last 10 minutes before the error is triggered in the logs, this was the test run before switching off VM. That test is still running and will provide its logs shortly, however, I am confident, it will show same results.
On 10/17/22 21:22, Granwille Strauss wrote:
Hi Wei
Please see attached. Please keep in mind, this worked 100% a few days ago when I tested on VM recurring snaps and it was perfect then. So I enabled recurring snaps for all 10 linux VMs I host. And I set them all to daily at 2AM.
They failed after the first day. So I figured maybe too many concurrent jobs running at the same time.
So I split them all to once weekly at different days at different hours. And still it failed. And now when doing it manually it fails too. But for some reason the snap ends up in /primary location is just returns an error for some reason.
On 10/17/22 21:13, Wei ZHOU wrote:
Hi,
What is the kvm host os and version ?
-Wei
On Monday, 17 October 2022, Granwille Strauss <gr...@namhost.com.invalid>
wrote:
The error seems to happen exactly at 1 hour intervals. All of them have
failed exactly at one hour. Also, these are all volume snapshots that I am
taking, so the logs I sent you so far are all in this regard. They are not
VM snaphots, nor are they powered off. I am running a new volume snap shot
again, this time VM powered off lets see.
I have also rebooted my entire KVM server and my management server. So far
zero luck.
On 10/17/22 20:26, Nicolas Vazquez wrote:
Can you try taking a manual volume snapshot and share both mgmt and agent logs please?
Regards,
Nicolas Vazquez
From: Granwille Strauss <gr...@namhost.com> <gr...@namhost.com>
Date: Monday, 17 October 2022 at 13:15
To: users@cloudstack.apache.org<ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org>
Cc: Nicolas Vazquez <Ni...@shapeblue.com> <Ni...@shapeblue.com>
Subject: Re: Recurring Snaps Failing
Hi
I also see another issue with snaps, I deleted all the failed snaps via the cloud management UI. But to my surprise, they're all still at /mnt/primary/snapshots/ see attached screenshot. Something odd is going on with snapshots.
On 10/17/22 17:35, Granwille Strauss wrote:
Hi
I am not 100% if I enabled debug correctly, I followed this https://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+agent+debug and ran
sed -i 's/INFO/DEBUG/g' /etc/cloudstack/agent/log4j-cloud.xml
Attached are the two log files. Same error. I reboot SSVM & vRouter prior no luck.
On 10/17/22 16:00, Granwille Strauss wrote:
Hi
No KVM is on its own server node and management on a different server node. However, management is is primary and secondary storage via NFS. I will enable DEBUG and re-run manual snap to replicate the error.
On 10/17/22 15:42, Nicolas Vazquez wrote:
Thanks, are the KVM agent and management server running on the same host? Can you increase the log level on the KVM agent logs to DEBUG and retry?
Regards,
Nicolas Vazquez
From: Granwille Strauss <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com>
Date: Monday, 17 October 2022 at 10:04
To: users@cloudstack.apache.org<ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org>
Cc: Nicolas Vazquez <Ni...@shapeblue.com> <Ni...@shapeblue.com> <Ni...@shapeblue.com>
Subject: Re: Recurring Snaps Failing
Please find attached TXT
On 10/17/22 14:54, Nicolas Vazquez wrote:
Ok, seems like the command is not reaching the KVM host, can you share the full management server logs for job-796?
Regards,
Nicolas Vazquez
From: Granwille Strauss <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com>
Date: Monday, 17 October 2022 at 09:32
To: users@cloudstack.apache.org<ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org>
Cc: Nicolas Vazquez <Ni...@shapeblue.com> <Ni...@shapeblue.com> <Ni...@shapeblue.com>
Subject: Re: Recurring Snaps Failing
Hi
This is all agent threw during the snap process nothing else:
2022-10-17 13:39:12,375 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:a864f9f6) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
2022-10-17 13:40:13,276 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:445216f3) Trying to fetch storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297 from libvirt
2022-10-17 13:40:13,289 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:445216f3) Asking libvirt to refresh storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297
2022-10-17 13:40:13,378 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:445216f3) Trying to fetch storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89 from libvirt
2022-10-17 13:40:13,393 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:445216f3) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
2022-10-17 13:41:13,950 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:87ecfee0) Trying to fetch storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297 from libvirt
2022-10-17 13:41:13,964 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:87ecfee0) Asking libvirt to refresh storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297
2022-10-17 13:41:14,054 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:87ecfee0) Trying to fetch storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89 from libvirt
2022-10-17 13:41:14,067 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:87ecfee0) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
2022-10-17 13:41:57,753 WARN [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-3:null) (logid:d91c35e5) Expected 1 answers while executing SetMonitorServiceCommand but received 3
2022-10-17 13:41:57,938 INFO [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:null) (logid:1549c9a1) Fetching health check result for SEVERIPADDRESS and executing fresh checks: false
On 10/17/22 14:29, Nicolas Vazquez wrote:
Hi Groete,
Can you share the CloudStack agent logs from the KVM host?
Regards,
Nicolas Vazquez
From: Granwille Strauss <gr...@namhost.com.INVALID> <gr...@namhost.com.INVALID> <gr...@namhost.com.INVALID> <gr...@namhost.com.INVALID> <gr...@namhost.com.INVALID>
Date: Monday, 17 October 2022 at 08:56
To: users@cloudstack.apache.org<ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org>
Subject: Re: Recurring Snaps Failing
Hi
I ran a few more tests and can now finally make sense of the failure:
2022-10-17 13:39:08,556 DEBUG [o.a.c.s.d.d.CloudStackPrimaryDataStoreDriverImpl] (Work-Job-Executor-1:ctx-93cb161b job-806/job-807 ctx-3f9ccf45) (logid:8fd10b20) Failed to take snapshot: 23
2022-10-17 13:39:08,558 DEBUG [o.a.c.s.s.SnapshotServiceImpl] (Work-Job-Executor-1:ctx-93cb161b job-806/job-807 ctx-3f9ccf45) (logid:8fd10b20) create snapshot Test001 failed: com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:1, com.cloud.exception.OperationTimedoutException: Commands 7160441932542378205 to Host 1 timed out after 3600
Seems to be a time out error. But there's no firewalls on the KVM host and NFS server to prevent connections and it worked fine before and randomly just stopped.
So I still have no idea where to solve this as all other necessary settings are in place. Any ideas?
On 10/17/22 08:51, Granwille Strauss wrote:
Hi Guys
I enabled recurring snap shots for my Linux VMs and they all seem to be failing with the following error:
root@hostname ~ $ grep error /var/log/cloudstack/management/management-server.log
2022-10-17 03:01:29,474 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-27:ctx-99d7d702 job-796 ctx-f26a81fe) (logid:f9a61b00) Failed to create snapshot due to an internal error creating snapshot for volume 635fd5b5-bf14-4cf4-979c-be8a649d5189
2022-10-17 03:01:29,476 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-27:ctx-99d7d702 job-796) (logid:f9a61b00) Complete async job-796, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 635fd5b5-bf14-4cf4-979c-be8a649d5189"}
2022-10-17 03:01:29,478 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-26:ctx-c7865967 job-794 ctx-0700afb0) (logid:5541113c) Failed to create snapshot due to an internal error creating snapshot for volume 70ffef79-3aad-43ab-93d1-153fcd34ad15
2022-10-17 03:01:29,480 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-26:ctx-c7865967 job-794) (logid:5541113c) Complete async job-794, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 70ffef79-3aad-43ab-93d1-153fcd34ad15"}
All I am getting out of this is internal error and nothing else. I have checked other logs and I am not quite sure where else to look.
Any assistance in this regard will be appreciated.
--
Regards / Groete
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
Granwille Strauss // Senior Systems Admin
e: granwille@namhost.com<ma...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com>
m: +264 81 323 1260<tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260>
w: www.namhost.com<http://www.namhost.com><http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com><http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com><http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
[https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
Namhost Internet Services (Pty) Ltd,
24 Black Eagle Rd, Hermanus <https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g><https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go to https://www.namhost.com/privacy-policy
[Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
--
Regards / Groete
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
Granwille Strauss // Senior Systems Admin
e: granwille@namhost.com<ma...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com>
m: +264 81 323 1260<tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260>
w: www.namhost.com<http://www.namhost.com><http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com><http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com><http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
[https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
Namhost Internet Services (Pty) Ltd,
24 Black Eagle Rd, Hermanus <https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g><https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go to https://www.namhost.com/privacy-policy
[Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
--
Regards / Groete
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
Granwille Strauss // Senior Systems Admin
e: granwille@namhost.com<ma...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com>
m: +264 81 323 1260<tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260>
w: www.namhost.com<http://www.namhost.com><http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
[https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
Namhost Internet Services (Pty) Ltd,
24 Black Eagle Rd, Hermanus <https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g><https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go to https://www.namhost.com/privacy-policy
<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
--
Regards / Groete
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
Granwille Strauss // Senior Systems Admin
e: granwille@namhost.com<ma...@namhost.com> <gr...@namhost.com>
m: +264 81 323 1260<tel:+264813231260><tel:+264813231260>
w: www.namhost.com<http://www.namhost.com><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
[https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
Namhost Internet Services (Pty) Ltd,
24 Black Eagle Rd, Hermanus <https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g><https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go to https://www.namhost.com/privacy-policy
[Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
--
Regards / Groete
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
Granwille Strauss // Senior Systems Admin
e: granwille@namhost.com<ma...@namhost.com> <gr...@namhost.com>
m: +264 81 323 1260<tel:+264813231260><tel:+264813231260>
w: www.namhost.com<http://www.namhost.com><https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/>
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
[https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
Namhost Internet Services (Pty) Ltd,
24 Black Eagle Rd, Hermanus <https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g><https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and s
--
Regards / Groete
<https://www.namhost.com><https://www.namhost.com> Granwille Strauss // Senior Systems Admin
*e:* granwille@namhost.com<ma...@namhost.com>
*m:* +264 81 323 1260 <+264813231260>
*w:* www.namhost.com<http://www.namhost.com>
<https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost>
<https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/>
<https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos>
<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
Namhost Internet Services (Pty) Ltd,
24 Black Eagle Rd, Hermanus <https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g><https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It is
forbidden to copy, forward, or in any way reveal the contents of this
message to anyone without our explicit consent. The integrity and security
of this email cannot be guaranteed over the Internet. Therefore, the sender
will not be held liable for any damage caused by the message. For our full
privacy policy and disclaimers, please go to https://www.namhost.com/
privacy-policy
[image: Powered by AdSigner]
<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
--
Regards / Groete
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
Granwille Strauss // Senior Systems Admin
e: granwille@namhost.com<ma...@namhost.com>
m: +264 81 323 1260<tel:+264813231260>
w: www.namhost.com<https://www.namhost.com/>
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
[https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
Namhost Internet Services (Pty) Ltd,
24 Black Eagle Rd, Hermanus, 7210, RSA
The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go to https://www.namhost.com/privacy-policy
[Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
--
Regards / Groete
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
Granwille Strauss // Senior Systems Admin
e: granwille@namhost.com<ma...@namhost.com>
m: +264 81 323 1260<tel:+264813231260>
w: www.namhost.com<https://www.namhost.com/>
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
[https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
Namhost Internet Services (Pty) Ltd,
24 Black Eagle Rd, Hermanus, 7210, RSA
The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go to https://www.namhost.com/privacy-policy
[Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
--
Regards / Groete
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
Granwille Strauss // Senior Systems Admin
e: granwille@namhost.com<ma...@namhost.com>
m: +264 81 323 1260<tel:+264813231260>
w: www.namhost.com<https://www.namhost.com/>
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
[https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
Namhost Internet Services (Pty) Ltd,
24 Black Eagle Rd, Hermanus, 7210, RSA
The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go to https://www.namhost.com/privacy-policy
[Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
--
Regards / Groete
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
Granwille Strauss // Senior Systems Admin
e: granwille@namhost.com<ma...@namhost.com>
m: +264 81 323 1260<tel:+264813231260>
w: www.namhost.com<https://www.namhost.com/>
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
[https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
Namhost Internet Services (Pty) Ltd,
24 Black Eagle Rd, Hermanus, 7210, RSA
The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go to https://www.namhost.com/privacy-policy
[Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
--
Regards / Groete
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
Granwille Strauss // Senior Systems Admin
e: granwille@namhost.com<ma...@namhost.com>
m: +264 81 323 1260<tel:+264813231260>
w: www.namhost.com<https://www.namhost.com/>
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
[https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
Namhost Internet Services (Pty) Ltd,
24 Black Eagle Rd, Hermanus, 7210, RSA
The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go to https://www.namhost.com/privacy-policy
[Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
--
Regards / Groete
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
Granwille Strauss // Senior Systems Admin
e: granwille@namhost.com<ma...@namhost.com>
m: +264 81 323 1260<tel:+264813231260>
w: www.namhost.com<https://www.namhost.com/>
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
[https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
Namhost Internet Services (Pty) Ltd,
24 Black Eagle Rd, Hermanus, 7210, RSA
The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go to https://www.namhost.com/privacy-policy
[Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
--
Regards / Groete
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
Granwille Strauss // Senior Systems Admin
e: granwille@namhost.com<ma...@namhost.com>
m: +264 81 323 1260<tel:+264813231260>
w: www.namhost.com<https://www.namhost.com/>
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
[https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
Namhost Internet Services (Pty) Ltd,
24 Black Eagle Rd, Hermanus, 7210, RSA
The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go to https://www.namhost.com/privacy-policy
[Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
--
Regards / Groete
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
Granwille Strauss // Senior Systems Admin
e: granwille@namhost.com<ma...@namhost.com>
m: +264 81 323 1260<tel:+264813231260>
w: www.namhost.com<https://www.namhost.com/>
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
[https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
Namhost Internet Services (Pty) Ltd,
24 Black Eagle Rd, Hermanus, 7210, RSA
The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go to https://www.namhost.com/privacy-policy
[Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
Re: Recurring Snaps Failing
Posted by Granwille Strauss <gr...@namhost.com.INVALID>.
Hi Guys
Was there anything else I could do to get this fixed? Should I report
this on Github? Also, is it possible to make recurring VM Snapshots and
not only Volume Snapshots?
On 10/18/22 11:43, Granwille Strauss wrote:
>
> Hi Wei
>
> Correct, yes VM Snapshot is 100% successful, its only volume snapshot
> that fails and its at the conversion it seems. As requested:
>
>> root@HOSTNAME ~ $ virsh domblklist i-2-19-VM
>> Target Source
>> ------------------------------------------------------------------------------------------
>> vda
>> /mnt/81ffca3a-9775-375d-a1c0-9504c0ec3d89/635fd5b5-bf14-4cf4-979c-be8a649d5189
>> hdc -
>>
>> root@HOSTNAME ~ $ qemu-img info
>> /mnt/81ffca3a-9775-375d-a1c0-9504c0ec3d89/635fd5b5-bf14-4cf4-979c-be8a649d5189
>> qemu-img: Could not open
>> '/mnt/81ffca3a-9775-375d-a1c0-9504c0ec3d89/635fd5b5-bf14-4cf4-979c-be8a649d5189':
>> Failed to get shared "write" lock
>> Is another process using the image
>> [/mnt/81ffca3a-9775-375d-a1c0-9504c0ec3d89/635fd5b5-bf14-4cf4-979c-be8a649d5189]?
>> root@athena ~ $
>
> Seems I cannot get you the details, however would this help?
>
>> root@HOSTNAME /mnt/81ffca3a-9775-375d-a1c0-9504c0ec3d89 $ du * -sh
>> 48G 340cc676-4683-4424-beb7-8b33a5cb3dfe
>> 42G 4a56546f-4f63-41c2-a314-27619fe5c2dc
>> 15G 635fd5b5-bf14-4cf4-979c-be8a649d5189
>> 21G 70ffef79-3aad-43ab-93d1-153fcd34ad15
>> 26G 777b5149-50c0-4c0d-944b-873050c85419
>> 20G 867a61e8-4635-40e4-b1b1-c94b2a325b49
>> 4.8G b21551c6-8276-4e58-841e-4e5a02c0e163
>> 11G d2c4feca-da5c-44b3-8ecc-40a01fd6c243
>> 40G f50c0bca-3ae2-48ee-aa9d-0e2d4ec6ba92
>> 24G f9336d09-0b3d-42d8-9a24-883b022b6712
>> 8.9G f962cb2c-e03b-4ed2-a035-1870a367aa46
>> 47G fd573618-885e-48c7-86e6-b019e2311d76
>> 8.0K KVMHA
>> 15G snapshots
>> root@HOSTNAME /mnt/81ffca3a-9775-375d-a1c0-9504c0ec3d89 $
>
> I have also added latest agent.log from today.
>
> On 10/18/22 11:07, Wei ZHOU wrote:
>> Hi,
>>
>> You can get the path of qcow2 image by "virsh domblklist i-2-19-VM".
>> The QCOW2 image does not have .qcow2 as a suffix.
>>
>> From what I see, a vm snapshot has been created 2022-10-18 09:44:21
>> +0200, which is good as taking the snapshot started at the same time.
>> so timed out when converting the vm snapshot to a qcow2 image (ons
>> secondary storage) by "qemu-img convert".
>>
>> ps: the agent.log you attached is out-of-date. Please share the
>> agent.log in few minutes after "2022-10-18 09:44:21"
>>
>>
>> -Wei
>>
>> On Tue, 18 Oct 2022 at 10:53, Granwille Strauss
>> <gr...@namhost.com> wrote:
>>
>> Hi Wei
>>
>> Thank you for the update. Thus far all the snapshot tests I took
>> were volume snaps and majority were when they are online and end
>> up failing. However, there's one VM where it seems to work just
>> fine, whether its online of offline and I am not sure what's
>> different from this VM as all other VM use the same Almalinux
>> template. Its just this one VM where it works as expected.
>>
>> In any case, I did a VM snapshot now with memory on my Ubuntu vm
>> while it was online and the VM Snap was successful, no issues
>> there. I ran a volume snap right after on the same ubuntu vm and
>> it failed again as before, see attached screenshot. So VM
>> snapshots seem to work fine, this tells me the issue is most
>> likely when its converted to QCOW2.
>>
>> In any case, please see answers to your questions below:
>>
>> 1. I am not finding any .qcow2 images on the KVM server to list
>> the QCOW2 size, or am I missing something?
>>
>>> root@HOSTNAME ~ $ find / -type f -iname '*.qcow2'
>>> root@HOSTNAME ~ $
>> As requested, here's a snapshot list of the current vm I am
>> testing on. Please let me know if you needed something specific?
>>
>>> root@HOSTNAME ~ $ virsh snapshot-list i-2-19-VM
>>> Name Creation
>>> Time State
>>> -----------------------------------------------------------------------------------
>>> 0843e6b1-cb12-495c-8d5f-35c9a3338db7 2022-10-17 18:41:08
>>> +0200 disk-snapshot
>>> 2e77acb7-1be4-4417-858e-a1d0890f44f0 2022-10-17 19:57:03
>>> +0200 disk-snapshot
>>> 68476e2a-aec8-4637-9c4b-2dc49d79bd82 2022-10-18 09:44:21
>>> +0200 disk-snapshot
>>> c110058b-4661-42c0-b648-a54f4ba4beb7 2022-10-17 22:34:21
>>> +0200 disk-snapshot
>>> i-2-19-VM_VS_20221018072924 2022-10-18 09:29:25
>>> +0200 running
>>
>> 2. Attached is a txt file from yesterdays agent.log. This is all
>> entries that happened between CopyCommand and CopyCmdAnswer.
>> There are zero errors, as you will see.
>>
>> 3. The 'backup.snapshot.wait' setting is currently at 21600
>> seconds = 6 hours, should this still be increased? Also, not sure
>> if 'copy.volume.wait' is relevant, but this is currently at 10800
>> seconds.
>>
>> On 10/18/22 09:30, Wei ZHOU wrote:
>>> Hi Granwille,
>>>
>>> The process of volume snapshot on running vms and stop vms are
>>> different. For running vms, cloudstack will first take a vm
>>> snapshot and convert the vm snapshot into a qcow2 image.
>>> It is normal that snapshotting a running vm takes longer time
>>> than a stopped vm, because
>>> (1) there is an extra step to take a vm snapshot. Depending on
>>> the memory size, the step might take a few minutes.
>>> (3) vm snapshot is saved in QCOW2 image (it is called Internal
>>> snapshots). When taking a vm snapshot, the QCOW2 image will
>>> enlarge. The image will not shrink when the vm snapshot is removed.
>>> If you have taken vm snapshot (or volume snapshot on running
>>> vms) multiple times, The QCOW2 image might be very big. The
>>> whole process (including vm snapshotting and qemu-img convert)
>>> might exceed the timeout limit (3600 seconds = 1 hour).
>>>
>>> can you please
>>> (1) check the QCOW2 image size and list the snapshot of QCOW2
>>> image (qemu-img info), list vm snapshot of the vm (virsh
>>> snapshot-list)
>>> (2) check the logs between CopyCommand and CopyCmdAnswer in
>>> agent.log
>>> (3) increase backup.snapshot.wait in global configuration and
>>> retry ("wait" in "CopyCommand" should be changed accordingly) ?
>>>
>>>
>>> -Wei
>>>
>>>
>>>
>>> On Tue, 18 Oct 2022 at 00:39, Granwille Strauss
>>> <gr...@namhost.com> wrote:
>>>
>>> Hi Guys
>>>
>>> I tested another Linux VM Almalinux VM, turned off the VM
>>> and made a successful snap. I turned it back online again
>>> and run another successful snap again. Now I run a snap on
>>> my Ubuntu VM and it failed. So just to be sure it wasn't
>>> just that one specific Almalinux VM that was working, I ran
>>> another snap on another Almalinux VM while it was online and
>>> too also failed. So I am not sure why recurring snaps is
>>> only working for this one VM and not the rest.
>>>
>>> Should I log this on GITHUB?
>>>
>>> On 10/17/22 21:55, Granwille Strauss wrote:
>>>>
>>>> Sorry, seems my attachments were too big, you can download
>>>> via Wetransfer: https://we.tl/t-9hUekbj86S
>>>>
>>>> And yes, I have the kvm.snapshot.enabled enabled in global
>>>> settings too, for live snapping.
>>>>
>>>> On 10/17/22 21:51, Granwille Strauss wrote:
>>>>>
>>>>> So it seems when VM is powered off, snap is successful.
>>>>> See attached logs for the successful snap, when VM is
>>>>> offline.
>>>>>
>>>>> On 10/17/22 21:38, Granwille Strauss wrote:
>>>>>>
>>>>>> Here's the last 10 minutes before the error is triggered
>>>>>> in the logs, this was the test run before switching off
>>>>>> VM. That test is still running and will provide its logs
>>>>>> shortly, however, I am confident, it will show same results.
>>>>>>
>>>>>> On 10/17/22 21:22, Granwille Strauss wrote:
>>>>>>>
>>>>>>> Hi Wei
>>>>>>>
>>>>>>> Please see attached. Please keep in mind, this worked
>>>>>>> 100% a few days ago when I tested on VM recurring snaps
>>>>>>> and it was perfect then. So I enabled recurring snaps
>>>>>>> for all 10 linux VMs I host. And I set them all to daily
>>>>>>> at 2AM.
>>>>>>> They failed after the first day. So I figured maybe too
>>>>>>> many concurrent jobs running at the same time.
>>>>>>>
>>>>>>> So I split them all to once weekly at different days at
>>>>>>> different hours. And still it failed. And now when doing
>>>>>>> it manually it fails too. But for some reason the snap
>>>>>>> ends up in /primary location is just returns an error
>>>>>>> for some reason.
>>>>>>>
>>>>>>> On 10/17/22 21:13, Wei ZHOU wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> What is the kvm host os and version ?
>>>>>>>>
>>>>>>>> -Wei
>>>>>>>>
>>>>>>>> On Monday, 17 October 2022, Granwille Strauss<gr...@namhost.com.invalid> <ma...@namhost.com.invalid>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> The error seems to happen exactly at 1 hour intervals. All of them have
>>>>>>>>> failed exactly at one hour. Also, these are all volume snapshots that I am
>>>>>>>>> taking, so the logs I sent you so far are all in this regard. They are not
>>>>>>>>> VM snaphots, nor are they powered off. I am running a new volume snap shot
>>>>>>>>> again, this time VM powered off lets see.
>>>>>>>>>
>>>>>>>>> I have also rebooted my entire KVM server and my management server. So far
>>>>>>>>> zero luck.
>>>>>>>>> On 10/17/22 20:26, Nicolas Vazquez wrote:
>>>>>>>>>
>>>>>>>>> Can you try taking a manual volume snapshot and share both mgmt and agent logs please?
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Nicolas Vazquez
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> From: Granwille Strauss<gr...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com>
>>>>>>>>> Date: Monday, 17 October 2022 at 13:15
>>>>>>>>> To:users@cloudstack.apache.org <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org>
>>>>>>>>> Cc: Nicolas Vazquez<Ni...@shapeblue.com> <ma...@shapeblue.com> <Ni...@shapeblue.com> <ma...@shapeblue.com>
>>>>>>>>> Subject: Re: Recurring Snaps Failing
>>>>>>>>>
>>>>>>>>> Hi
>>>>>>>>>
>>>>>>>>> I also see another issue with snaps, I deleted all the failed snaps via the cloud management UI. But to my surprise, they're all still at /mnt/primary/snapshots/ see attached screenshot. Something odd is going on with snapshots.
>>>>>>>>> On 10/17/22 17:35, Granwille Strauss wrote:
>>>>>>>>>
>>>>>>>>> Hi
>>>>>>>>>
>>>>>>>>> I am not 100% if I enabled debug correctly, I followed thishttps://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+agent+debug and ran
>>>>>>>>> sed -i 's/INFO/DEBUG/g' /etc/cloudstack/agent/log4j-cloud.xml
>>>>>>>>>
>>>>>>>>> Attached are the two log files. Same error. I reboot SSVM & vRouter prior no luck.
>>>>>>>>> On 10/17/22 16:00, Granwille Strauss wrote:
>>>>>>>>>
>>>>>>>>> Hi
>>>>>>>>>
>>>>>>>>> No KVM is on its own server node and management on a different server node. However, management is is primary and secondary storage via NFS. I will enable DEBUG and re-run manual snap to replicate the error.
>>>>>>>>> On 10/17/22 15:42, Nicolas Vazquez wrote:
>>>>>>>>> Thanks, are the KVM agent and management server running on the same host? Can you increase the log level on the KVM agent logs to DEBUG and retry?
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Nicolas Vazquez
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> From: Granwille Strauss<gr...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com>
>>>>>>>>> Date: Monday, 17 October 2022 at 10:04
>>>>>>>>> To:users@cloudstack.apache.org<ma...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org>
>>>>>>>>> Cc: Nicolas Vazquez<Ni...@shapeblue.com> <ma...@shapeblue.com> <Ni...@shapeblue.com> <ma...@shapeblue.com> <ma...@shapeblue.com> <Ni...@shapeblue.com> <ma...@shapeblue.com>
>>>>>>>>> Subject: Re: Recurring Snaps Failing
>>>>>>>>>
>>>>>>>>> Please find attached TXT
>>>>>>>>> On 10/17/22 14:54, Nicolas Vazquez wrote:
>>>>>>>>>
>>>>>>>>> Ok, seems like the command is not reaching the KVM host, can you share the full management server logs for job-796?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>> Nicolas Vazquez
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> From: Granwille Strauss<gr...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com>
>>>>>>>>>
>>>>>>>>> Date: Monday, 17 October 2022 at 09:32
>>>>>>>>>
>>>>>>>>> To:users@cloudstack.apache.org<ma...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org>
>>>>>>>>>
>>>>>>>>> Cc: Nicolas Vazquez<Ni...@shapeblue.com> <ma...@shapeblue.com> <Ni...@shapeblue.com> <ma...@shapeblue.com> <ma...@shapeblue.com> <Ni...@shapeblue.com> <ma...@shapeblue.com>
>>>>>>>>>
>>>>>>>>> Subject: Re: Recurring Snaps Failing
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> This is all agent threw during the snap process nothing else:
>>>>>>>>>
>>>>>>>>> 2022-10-17 13:39:12,375 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:a864f9f6) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
>>>>>>>>>
>>>>>>>>> 2022-10-17 13:40:13,276 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:445216f3) Trying to fetch storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297 from libvirt
>>>>>>>>>
>>>>>>>>> 2022-10-17 13:40:13,289 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:445216f3) Asking libvirt to refresh storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297
>>>>>>>>>
>>>>>>>>> 2022-10-17 13:40:13,378 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:445216f3) Trying to fetch storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89 from libvirt
>>>>>>>>>
>>>>>>>>> 2022-10-17 13:40:13,393 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:445216f3) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
>>>>>>>>>
>>>>>>>>> 2022-10-17 13:41:13,950 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:87ecfee0) Trying to fetch storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297 from libvirt
>>>>>>>>>
>>>>>>>>> 2022-10-17 13:41:13,964 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:87ecfee0) Asking libvirt to refresh storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297
>>>>>>>>>
>>>>>>>>> 2022-10-17 13:41:14,054 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:87ecfee0) Trying to fetch storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89 from libvirt
>>>>>>>>>
>>>>>>>>> 2022-10-17 13:41:14,067 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:87ecfee0) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
>>>>>>>>>
>>>>>>>>> 2022-10-17 13:41:57,753 WARN [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-3:null) (logid:d91c35e5) Expected 1 answers while executing SetMonitorServiceCommand but received 3
>>>>>>>>>
>>>>>>>>> 2022-10-17 13:41:57,938 INFO [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:null) (logid:1549c9a1) Fetching health check result for SEVERIPADDRESS and executing fresh checks: false
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 10/17/22 14:29, Nicolas Vazquez wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi Groete,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Can you share the CloudStack agent logs from the KVM host?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Nicolas Vazquez
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> From: Granwille Strauss<gr...@namhost.com.INVALID> <ma...@namhost.com.INVALID> <gr...@namhost.com.INVALID> <ma...@namhost.com.INVALID> <ma...@namhost.com.INVALID> <gr...@namhost.com.INVALID> <ma...@namhost.com.INVALID> <ma...@namhost.com.INVALID> <gr...@namhost.com.INVALID> <ma...@namhost.com.INVALID> <ma...@namhost.com.INVALID> <gr...@namhost.com.INVALID> <ma...@namhost.com.INVALID>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Date: Monday, 17 October 2022 at 08:56
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> To:users@cloudstack.apache.org<ma...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Subject: Re: Recurring Snaps Failing
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I ran a few more tests and can now finally make sense of the failure:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2022-10-17 13:39:08,556 DEBUG [o.a.c.s.d.d.CloudStackPrimaryDataStoreDriverImpl] (Work-Job-Executor-1:ctx-93cb161b job-806/job-807 ctx-3f9ccf45) (logid:8fd10b20) Failed to take snapshot: 23
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2022-10-17 13:39:08,558 DEBUG [o.a.c.s.s.SnapshotServiceImpl] (Work-Job-Executor-1:ctx-93cb161b job-806/job-807 ctx-3f9ccf45) (logid:8fd10b20) create snapshot Test001 failed: com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:1, com.cloud.exception.OperationTimedoutException: Commands 7160441932542378205 to Host 1 timed out after 3600
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Seems to be a time out error. But there's no firewalls on the KVM host and NFS server to prevent connections and it worked fine before and randomly just stopped.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> So I still have no idea where to solve this as all other necessary settings are in place. Any ideas?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 10/17/22 08:51, Granwille Strauss wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi Guys
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I enabled recurring snap shots for my Linux VMs and they all seem to be failing with the following error:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> root@hostname ~ $ grep error /var/log/cloudstack/management/management-server.log
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2022-10-17 03:01:29,474 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-27:ctx-99d7d702 job-796 ctx-f26a81fe) (logid:f9a61b00) Failed to create snapshot due to an internal error creating snapshot for volume 635fd5b5-bf14-4cf4-979c-be8a649d5189
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2022-10-17 03:01:29,476 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-27:ctx-99d7d702 job-796) (logid:f9a61b00) Complete async job-796, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 635fd5b5-bf14-4cf4-979c-be8a649d5189"}
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2022-10-17 03:01:29,478 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-26:ctx-c7865967 job-794 ctx-0700afb0) (logid:5541113c) Failed to create snapshot due to an internal error creating snapshot for volume 70ffef79-3aad-43ab-93d1-153fcd34ad15
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2022-10-17 03:01:29,480 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-26:ctx-c7865967 job-794) (logid:5541113c) Complete async job-794, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 70ffef79-3aad-43ab-93d1-153fcd34ad15"}
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> All I am getting out of this is internal error and nothing else. I have checked other logs and I am not quite sure where else to look.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Any assistance in this regard will be appreciated.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Regards / Groete
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Granwille Strauss // Senior Systems Admin
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> e:granwille@namhost.com<ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> m: +264 81 323 1260<tel:+264813231260> <tel:+264813231260><tel:+264813231260> <tel:+264813231260><tel:+264813231260> <tel:+264813231260><tel:+264813231260> <tel:+264813231260>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> w:www.namhost.com <http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com> <http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com> <http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com> <http://www.namhost.com> <http://www.namhost.com><https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Namhost Internet Services (Pty) Ltd,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g> <https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Regards / Groete
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Granwille Strauss // Senior Systems Admin
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> e:granwille@namhost.com<ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> m: +264 81 323 1260<tel:+264813231260> <tel:+264813231260><tel:+264813231260> <tel:+264813231260><tel:+264813231260> <tel:+264813231260><tel:+264813231260> <tel:+264813231260>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> w:www.namhost.com <http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com> <http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com> <http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com> <http://www.namhost.com> <http://www.namhost.com><https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Namhost Internet Services (Pty) Ltd,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g> <https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Regards / Groete
>>>>>>>>>
>>>>>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/>
>>>>>>>>>
>>>>>>>>> Granwille Strauss // Senior Systems Admin
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> e:granwille@namhost.com<ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com>
>>>>>>>>>
>>>>>>>>> m: +264 81 323 1260<tel:+264813231260> <tel:+264813231260><tel:+264813231260> <tel:+264813231260>
>>>>>>>>>
>>>>>>>>> w:www.namhost.com <http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com> <http://www.namhost.com> <http://www.namhost.com><https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>>>>>>
>>>>>>>>> Namhost Internet Services (Pty) Ltd,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g> <https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Regards / Groete
>>>>>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/>
>>>>>>>>> Granwille Strauss // Senior Systems Admin
>>>>>>>>>
>>>>>>>>> e:granwille@namhost.com<ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com>
>>>>>>>>> m: +264 81 323 1260<tel:+264813231260> <tel:+264813231260>
>>>>>>>>> w:www.namhost.com <http://www.namhost.com><https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/>
>>>>>>>>>
>>>>>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>>>>>>
>>>>>>>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>>>>>> Namhost Internet Services (Pty) Ltd,
>>>>>>>>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g> <https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>>>>>>>>
>>>>>>>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>>>>>>>
>>>>>>>>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>>>>>>> --
>>>>>>>>>
>>>>>>>>> Regards / Groete
>>>>>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/>
>>>>>>>>> Granwille Strauss // Senior Systems Admin
>>>>>>>>>
>>>>>>>>> e:granwille@namhost.com<ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com>
>>>>>>>>> m: +264 81 323 1260<tel:+264813231260> <tel:+264813231260>
>>>>>>>>> w:www.namhost.com <http://www.namhost.com><https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/>
>>>>>>>>>
>>>>>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>>>>>>
>>>>>>>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>>>>>> Namhost Internet Services (Pty) Ltd,
>>>>>>>>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g> <https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>>>>>>>>
>>>>>>>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and s
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Regards / Groete
>>>>>>>>>
>>>>>>>>> <https://www.namhost.com> <https://www.namhost.com> Granwille Strauss // Senior Systems Admin
>>>>>>>>>
>>>>>>>>> *e:*granwille@namhost.com
>>>>>>>>> *m:* +264 81 323 1260 <+264813231260>
>>>>>>>>> *w:*www.namhost.com <http://www.namhost.com>
>>>>>>>>>
>>>>>>>>> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost>
>>>>>>>>> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/>
>>>>>>>>> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos>
>>>>>>>>> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>>>>>>
>>>>>>>>> Namhost Internet Services (Pty) Ltd,
>>>>>>>>>
>>>>>>>>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g> <https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> The content of this message is confidential. If you have received it by
>>>>>>>>> mistake, please inform us by email reply and then delete the message. It is
>>>>>>>>> forbidden to copy, forward, or in any way reveal the contents of this
>>>>>>>>> message to anyone without our explicit consent. The integrity and security
>>>>>>>>> of this email cannot be guaranteed over the Internet. Therefore, the sender
>>>>>>>>> will not be held liable for any damage caused by the message. For our full
>>>>>>>>> privacy policy and disclaimers, please go tohttps://www.namhost.com/
>>>>>>>>> privacy-policy
>>>>>>>>>
>>>>>>>>> [image: Powered by AdSigner]
>>>>>>>>> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>>>>>>>
>>>>>>> --
>>>>>>> Regards / Groete
>>>>>>>
>>>>>>> <https://www.namhost.com> Granwille Strauss // Senior
>>>>>>> Systems Admin
>>>>>>>
>>>>>>> *e:* granwille@namhost.com
>>>>>>> *m:* +264 81 323 1260 <tel:+264813231260>
>>>>>>> *w:* www.namhost.com <https://www.namhost.com/>
>>>>>>>
>>>>>>> <https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>>>>
>>>>>>> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>>>>
>>>>>>> Namhost Internet Services (Pty) Ltd,
>>>>>>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>>>>>>
>>>>>>>
>>>>>>> The content of this message is confidential. If you have
>>>>>>> received it by mistake, please inform us by email reply
>>>>>>> and then delete the message. It is forbidden to copy,
>>>>>>> forward, or in any way reveal the contents of this
>>>>>>> message to anyone without our explicit consent. The
>>>>>>> integrity and security of this email cannot be
>>>>>>> guaranteed over the Internet. Therefore, the sender will
>>>>>>> not be held liable for any damage caused by the message.
>>>>>>> For our full privacy policy and disclaimers, please go
>>>>>>> to https://www.namhost.com/privacy-policy
>>>>>>>
>>>>>>> Powered by AdSigner
>>>>>>> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>>>> --
>>>>>> Regards / Groete
>>>>>>
>>>>>> <https://www.namhost.com> Granwille Strauss // Senior
>>>>>> Systems Admin
>>>>>>
>>>>>> *e:* granwille@namhost.com
>>>>>> *m:* +264 81 323 1260 <tel:+264813231260>
>>>>>> *w:* www.namhost.com <https://www.namhost.com/>
>>>>>>
>>>>>> <https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>>>
>>>>>> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>>>
>>>>>> Namhost Internet Services (Pty) Ltd,
>>>>>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>>>>>
>>>>>>
>>>>>> The content of this message is confidential. If you have
>>>>>> received it by mistake, please inform us by email reply
>>>>>> and then delete the message. It is forbidden to copy,
>>>>>> forward, or in any way reveal the contents of this
>>>>>> message to anyone without our explicit consent. The
>>>>>> integrity and security of this email cannot be guaranteed
>>>>>> over the Internet. Therefore, the sender will not be held
>>>>>> liable for any damage caused by the message. For our full
>>>>>> privacy policy and disclaimers, please go to
>>>>>> https://www.namhost.com/privacy-policy
>>>>>>
>>>>>> Powered by AdSigner
>>>>>> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>>> --
>>>>> Regards / Groete
>>>>>
>>>>> <https://www.namhost.com> Granwille Strauss // Senior
>>>>> Systems Admin
>>>>>
>>>>> *e:* granwille@namhost.com
>>>>> *m:* +264 81 323 1260 <tel:+264813231260>
>>>>> *w:* www.namhost.com <https://www.namhost.com/>
>>>>>
>>>>> <https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>>
>>>>> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>>
>>>>> Namhost Internet Services (Pty) Ltd,
>>>>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>>>>
>>>>>
>>>>> The content of this message is confidential. If you have
>>>>> received it by mistake, please inform us by email reply
>>>>> and then delete the message. It is forbidden to copy,
>>>>> forward, or in any way reveal the contents of this message
>>>>> to anyone without our explicit consent. The integrity and
>>>>> security of this email cannot be guaranteed over the
>>>>> Internet. Therefore, the sender will not be held liable
>>>>> for any damage caused by the message. For our full privacy
>>>>> policy and disclaimers, please go to
>>>>> https://www.namhost.com/privacy-policy
>>>>>
>>>>> Powered by AdSigner
>>>>> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>> --
>>>> Regards / Groete
>>>>
>>>> <https://www.namhost.com> Granwille Strauss // Senior
>>>> Systems Admin
>>>>
>>>> *e:* granwille@namhost.com
>>>> *m:* +264 81 323 1260 <tel:+264813231260>
>>>> *w:* www.namhost.com <https://www.namhost.com/>
>>>>
>>>> <https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>
>>>> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>
>>>> Namhost Internet Services (Pty) Ltd,
>>>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>>>
>>>>
>>>> The content of this message is confidential. If you have
>>>> received it by mistake, please inform us by email reply and
>>>> then delete the message. It is forbidden to copy, forward,
>>>> or in any way reveal the contents of this message to anyone
>>>> without our explicit consent. The integrity and security of
>>>> this email cannot be guaranteed over the Internet.
>>>> Therefore, the sender will not be held liable for any
>>>> damage caused by the message. For our full privacy policy
>>>> and disclaimers, please go to
>>>> https://www.namhost.com/privacy-policy
>>>>
>>>> Powered by AdSigner
>>>> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>> --
>>> Regards / Groete
>>>
>>> <https://www.namhost.com> Granwille Strauss // Senior
>>> Systems Admin
>>>
>>> *e:* granwille@namhost.com
>>> *m:* +264 81 323 1260 <tel:+264813231260>
>>> *w:* www.namhost.com <https://www.namhost.com/>
>>>
>>> <https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>
>>> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>
>>> Namhost Internet Services (Pty) Ltd,
>>>
>>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>>
>>>
>>>
>>> The content of this message is confidential. If you have
>>> received it by mistake, please inform us by email reply and
>>> then delete the message. It is forbidden to copy, forward,
>>> or in any way reveal the contents of this message to anyone
>>> without our explicit consent. The integrity and security of
>>> this email cannot be guaranteed over the Internet.
>>> Therefore, the sender will not be held liable for any damage
>>> caused by the message. For our full privacy policy and
>>> disclaimers, please go to
>>> https://www.namhost.com/privacy-policy
>>>
>>> Powered by AdSigner
>>> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>
>> --
>> Regards / Groete
>>
>> <https://www.namhost.com> Granwille Strauss // Senior Systems Admin
>>
>> *e:* granwille@namhost.com
>> *m:* +264 81 323 1260 <tel:+264813231260>
>> *w:* www.namhost.com <https://www.namhost.com/>
>>
>> <https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>
>> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>
>> Namhost Internet Services (Pty) Ltd,
>>
>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>
>>
>>
>> The content of this message is confidential. If you have received
>> it by mistake, please inform us by email reply and then delete
>> the message. It is forbidden to copy, forward, or in any way
>> reveal the contents of this message to anyone without our
>> explicit consent. The integrity and security of this email cannot
>> be guaranteed over the Internet. Therefore, the sender will not
>> be held liable for any damage caused by the message. For our full
>> privacy policy and disclaimers, please go to
>> https://www.namhost.com/privacy-policy
>>
>> Powered by AdSigner
>> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>
> --
> Regards / Groete
>
> <https://www.namhost.com> Granwille Strauss // Senior Systems Admin
>
> *e:* granwille@namhost.com
> *m:* +264 81 323 1260 <tel:+264813231260>
> *w:* www.namhost.com <https://www.namhost.com/>
>
> <https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>
> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>
> Namhost Internet Services (Pty) Ltd,
> 24 Black Eagle Rd, Hermanus, 7210, RSA
>
>
> The content of this message is confidential. If you have received it
> by mistake, please inform us by email reply and then delete the
> message. It is forbidden to copy, forward, or in any way reveal the
> contents of this message to anyone without our explicit consent. The
> integrity and security of this email cannot be guaranteed over the
> Internet. Therefore, the sender will not be held liable for any damage
> caused by the message. For our full privacy policy and disclaimers,
> please go to https://www.namhost.com/privacy-policy
>
> Powered by AdSigner
> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
--
Regards / Groete
<https://www.namhost.com> Granwille Strauss // Senior Systems Admin
*e:* granwille@namhost.com
*m:* +264 81 323 1260 <tel:+264813231260>
*w:* www.namhost.com <https://www.namhost.com/>
<https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
Namhost Internet Services (Pty) Ltd,
24 Black Eagle Rd, Hermanus, 7210, RSA
The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It
is forbidden to copy, forward, or in any way reveal the contents of this
message to anyone without our explicit consent. The integrity and
security of this email cannot be guaranteed over the Internet.
Therefore, the sender will not be held liable for any damage caused by
the message. For our full privacy policy and disclaimers, please go to
https://www.namhost.com/privacy-policy
Powered by AdSigner
<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
Re: Recurring Snaps Failing
Posted by Granwille Strauss <gr...@namhost.com.INVALID>.
Hi Wei
Correct, yes VM Snapshot is 100% successful, its only volume snapshot
that fails and its at the conversion it seems. As requested:
> root@HOSTNAME ~ $ virsh domblklist i-2-19-VM
> Target Source
> ------------------------------------------------------------------------------------------
> vda
> /mnt/81ffca3a-9775-375d-a1c0-9504c0ec3d89/635fd5b5-bf14-4cf4-979c-be8a649d5189
> hdc -
>
> root@HOSTNAME ~ $ qemu-img info
> /mnt/81ffca3a-9775-375d-a1c0-9504c0ec3d89/635fd5b5-bf14-4cf4-979c-be8a649d5189
> qemu-img: Could not open
> '/mnt/81ffca3a-9775-375d-a1c0-9504c0ec3d89/635fd5b5-bf14-4cf4-979c-be8a649d5189':
> Failed to get shared "write" lock
> Is another process using the image
> [/mnt/81ffca3a-9775-375d-a1c0-9504c0ec3d89/635fd5b5-bf14-4cf4-979c-be8a649d5189]?
> root@athena ~ $
Seems I cannot get you the details, however would this help?
> root@HOSTNAME /mnt/81ffca3a-9775-375d-a1c0-9504c0ec3d89 $ du * -sh
> 48G 340cc676-4683-4424-beb7-8b33a5cb3dfe
> 42G 4a56546f-4f63-41c2-a314-27619fe5c2dc
> 15G 635fd5b5-bf14-4cf4-979c-be8a649d5189
> 21G 70ffef79-3aad-43ab-93d1-153fcd34ad15
> 26G 777b5149-50c0-4c0d-944b-873050c85419
> 20G 867a61e8-4635-40e4-b1b1-c94b2a325b49
> 4.8G b21551c6-8276-4e58-841e-4e5a02c0e163
> 11G d2c4feca-da5c-44b3-8ecc-40a01fd6c243
> 40G f50c0bca-3ae2-48ee-aa9d-0e2d4ec6ba92
> 24G f9336d09-0b3d-42d8-9a24-883b022b6712
> 8.9G f962cb2c-e03b-4ed2-a035-1870a367aa46
> 47G fd573618-885e-48c7-86e6-b019e2311d76
> 8.0K KVMHA
> 15G snapshots
> root@HOSTNAME /mnt/81ffca3a-9775-375d-a1c0-9504c0ec3d89 $
I have also added latest agent.log from today.
On 10/18/22 11:07, Wei ZHOU wrote:
> Hi,
>
> You can get the path of qcow2 image by "virsh domblklist i-2-19-VM".
> The QCOW2 image does not have .qcow2 as a suffix.
>
> From what I see, a vm snapshot has been created 2022-10-18 09:44:21
> +0200, which is good as taking the snapshot started at the same time.
> so timed out when converting the vm snapshot to a qcow2 image (ons
> secondary storage) by "qemu-img convert".
>
> ps: the agent.log you attached is out-of-date. Please share the
> agent.log in few minutes after "2022-10-18 09:44:21"
>
>
> -Wei
>
> On Tue, 18 Oct 2022 at 10:53, Granwille Strauss
> <gr...@namhost.com> wrote:
>
> Hi Wei
>
> Thank you for the update. Thus far all the snapshot tests I took
> were volume snaps and majority were when they are online and end
> up failing. However, there's one VM where it seems to work just
> fine, whether its online of offline and I am not sure what's
> different from this VM as all other VM use the same Almalinux
> template. Its just this one VM where it works as expected.
>
> In any case, I did a VM snapshot now with memory on my Ubuntu vm
> while it was online and the VM Snap was successful, no issues
> there. I ran a volume snap right after on the same ubuntu vm and
> it failed again as before, see attached screenshot. So VM
> snapshots seem to work fine, this tells me the issue is most
> likely when its converted to QCOW2.
>
> In any case, please see answers to your questions below:
>
> 1. I am not finding any .qcow2 images on the KVM server to list
> the QCOW2 size, or am I missing something?
>
>> root@HOSTNAME ~ $ find / -type f -iname '*.qcow2'
>> root@HOSTNAME ~ $
> As requested, here's a snapshot list of the current vm I am
> testing on. Please let me know if you needed something specific?
>
>> root@HOSTNAME ~ $ virsh snapshot-list i-2-19-VM
>> Name Creation
>> Time State
>> -----------------------------------------------------------------------------------
>> 0843e6b1-cb12-495c-8d5f-35c9a3338db7 2022-10-17 18:41:08
>> +0200 disk-snapshot
>> 2e77acb7-1be4-4417-858e-a1d0890f44f0 2022-10-17 19:57:03
>> +0200 disk-snapshot
>> 68476e2a-aec8-4637-9c4b-2dc49d79bd82 2022-10-18 09:44:21
>> +0200 disk-snapshot
>> c110058b-4661-42c0-b648-a54f4ba4beb7 2022-10-17 22:34:21
>> +0200 disk-snapshot
>> i-2-19-VM_VS_20221018072924 2022-10-18 09:29:25
>> +0200 running
>
> 2. Attached is a txt file from yesterdays agent.log. This is all
> entries that happened between CopyCommand and CopyCmdAnswer. There
> are zero errors, as you will see.
>
> 3. The 'backup.snapshot.wait' setting is currently at 21600
> seconds = 6 hours, should this still be increased? Also, not sure
> if 'copy.volume.wait' is relevant, but this is currently at 10800
> seconds.
>
> On 10/18/22 09:30, Wei ZHOU wrote:
>> Hi Granwille,
>>
>> The process of volume snapshot on running vms and stop vms are
>> different. For running vms, cloudstack will first take a vm
>> snapshot and convert the vm snapshot into a qcow2 image.
>> It is normal that snapshotting a running vm takes longer time
>> than a stopped vm, because
>> (1) there is an extra step to take a vm snapshot. Depending on
>> the memory size, the step might take a few minutes.
>> (3) vm snapshot is saved in QCOW2 image (it is called Internal
>> snapshots). When taking a vm snapshot, the QCOW2 image will
>> enlarge. The image will not shrink when the vm snapshot is removed.
>> If you have taken vm snapshot (or volume snapshot on running vms)
>> multiple times, The QCOW2 image might be very big. The
>> whole process (including vm snapshotting and qemu-img convert)
>> might exceed the timeout limit (3600 seconds = 1 hour).
>>
>> can you please
>> (1) check the QCOW2 image size and list the snapshot of QCOW2
>> image (qemu-img info), list vm snapshot of the vm (virsh
>> snapshot-list)
>> (2) check the logs between CopyCommand and CopyCmdAnswer in agent.log
>> (3) increase backup.snapshot.wait in global configuration and
>> retry ("wait" in "CopyCommand" should be changed accordingly) ?
>>
>>
>> -Wei
>>
>>
>>
>> On Tue, 18 Oct 2022 at 00:39, Granwille Strauss
>> <gr...@namhost.com> wrote:
>>
>> Hi Guys
>>
>> I tested another Linux VM Almalinux VM, turned off the VM and
>> made a successful snap. I turned it back online again and run
>> another successful snap again. Now I run a snap on my Ubuntu
>> VM and it failed. So just to be sure it wasn't just that one
>> specific Almalinux VM that was working, I ran another snap on
>> another Almalinux VM while it was online and too also failed.
>> So I am not sure why recurring snaps is only working for this
>> one VM and not the rest.
>>
>> Should I log this on GITHUB?
>>
>> On 10/17/22 21:55, Granwille Strauss wrote:
>>>
>>> Sorry, seems my attachments were too big, you can download
>>> via Wetransfer: https://we.tl/t-9hUekbj86S
>>>
>>> And yes, I have the kvm.snapshot.enabled enabled in global
>>> settings too, for live snapping.
>>>
>>> On 10/17/22 21:51, Granwille Strauss wrote:
>>>>
>>>> So it seems when VM is powered off, snap is successful. See
>>>> attached logs for the successful snap, when VM is offline.
>>>>
>>>> On 10/17/22 21:38, Granwille Strauss wrote:
>>>>>
>>>>> Here's the last 10 minutes before the error is triggered
>>>>> in the logs, this was the test run before switching off
>>>>> VM. That test is still running and will provide its logs
>>>>> shortly, however, I am confident, it will show same results.
>>>>>
>>>>> On 10/17/22 21:22, Granwille Strauss wrote:
>>>>>>
>>>>>> Hi Wei
>>>>>>
>>>>>> Please see attached. Please keep in mind, this worked
>>>>>> 100% a few days ago when I tested on VM recurring snaps
>>>>>> and it was perfect then. So I enabled recurring snaps for
>>>>>> all 10 linux VMs I host. And I set them all to daily at 2AM.
>>>>>> They failed after the first day. So I figured maybe too
>>>>>> many concurrent jobs running at the same time.
>>>>>>
>>>>>> So I split them all to once weekly at different days at
>>>>>> different hours. And still it failed. And now when doing
>>>>>> it manually it fails too. But for some reason the snap
>>>>>> ends up in /primary location is just returns an error for
>>>>>> some reason.
>>>>>>
>>>>>> On 10/17/22 21:13, Wei ZHOU wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> What is the kvm host os and version ?
>>>>>>>
>>>>>>> -Wei
>>>>>>>
>>>>>>> On Monday, 17 October 2022, Granwille Strauss<gr...@namhost.com.invalid> <ma...@namhost.com.invalid>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> The error seems to happen exactly at 1 hour intervals. All of them have
>>>>>>>> failed exactly at one hour. Also, these are all volume snapshots that I am
>>>>>>>> taking, so the logs I sent you so far are all in this regard. They are not
>>>>>>>> VM snaphots, nor are they powered off. I am running a new volume snap shot
>>>>>>>> again, this time VM powered off lets see.
>>>>>>>>
>>>>>>>> I have also rebooted my entire KVM server and my management server. So far
>>>>>>>> zero luck.
>>>>>>>> On 10/17/22 20:26, Nicolas Vazquez wrote:
>>>>>>>>
>>>>>>>> Can you try taking a manual volume snapshot and share both mgmt and agent logs please?
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Nicolas Vazquez
>>>>>>>>
>>>>>>>>
>>>>>>>> From: Granwille Strauss<gr...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com>
>>>>>>>> Date: Monday, 17 October 2022 at 13:15
>>>>>>>> To:users@cloudstack.apache.org <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org>
>>>>>>>> Cc: Nicolas Vazquez<Ni...@shapeblue.com> <ma...@shapeblue.com> <Ni...@shapeblue.com> <ma...@shapeblue.com>
>>>>>>>> Subject: Re: Recurring Snaps Failing
>>>>>>>>
>>>>>>>> Hi
>>>>>>>>
>>>>>>>> I also see another issue with snaps, I deleted all the failed snaps via the cloud management UI. But to my surprise, they're all still at /mnt/primary/snapshots/ see attached screenshot. Something odd is going on with snapshots.
>>>>>>>> On 10/17/22 17:35, Granwille Strauss wrote:
>>>>>>>>
>>>>>>>> Hi
>>>>>>>>
>>>>>>>> I am not 100% if I enabled debug correctly, I followed thishttps://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+agent+debug and ran
>>>>>>>> sed -i 's/INFO/DEBUG/g' /etc/cloudstack/agent/log4j-cloud.xml
>>>>>>>>
>>>>>>>> Attached are the two log files. Same error. I reboot SSVM & vRouter prior no luck.
>>>>>>>> On 10/17/22 16:00, Granwille Strauss wrote:
>>>>>>>>
>>>>>>>> Hi
>>>>>>>>
>>>>>>>> No KVM is on its own server node and management on a different server node. However, management is is primary and secondary storage via NFS. I will enable DEBUG and re-run manual snap to replicate the error.
>>>>>>>> On 10/17/22 15:42, Nicolas Vazquez wrote:
>>>>>>>> Thanks, are the KVM agent and management server running on the same host? Can you increase the log level on the KVM agent logs to DEBUG and retry?
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Nicolas Vazquez
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> From: Granwille Strauss<gr...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com>
>>>>>>>> Date: Monday, 17 October 2022 at 10:04
>>>>>>>> To:users@cloudstack.apache.org<ma...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org>
>>>>>>>> Cc: Nicolas Vazquez<Ni...@shapeblue.com> <ma...@shapeblue.com> <Ni...@shapeblue.com> <ma...@shapeblue.com> <ma...@shapeblue.com> <Ni...@shapeblue.com> <ma...@shapeblue.com>
>>>>>>>> Subject: Re: Recurring Snaps Failing
>>>>>>>>
>>>>>>>> Please find attached TXT
>>>>>>>> On 10/17/22 14:54, Nicolas Vazquez wrote:
>>>>>>>>
>>>>>>>> Ok, seems like the command is not reaching the KVM host, can you share the full management server logs for job-796?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>> Nicolas Vazquez
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> From: Granwille Strauss<gr...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com>
>>>>>>>>
>>>>>>>> Date: Monday, 17 October 2022 at 09:32
>>>>>>>>
>>>>>>>> To:users@cloudstack.apache.org<ma...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org>
>>>>>>>>
>>>>>>>> Cc: Nicolas Vazquez<Ni...@shapeblue.com> <ma...@shapeblue.com> <Ni...@shapeblue.com> <ma...@shapeblue.com> <ma...@shapeblue.com> <Ni...@shapeblue.com> <ma...@shapeblue.com>
>>>>>>>>
>>>>>>>> Subject: Re: Recurring Snaps Failing
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Hi
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> This is all agent threw during the snap process nothing else:
>>>>>>>>
>>>>>>>> 2022-10-17 13:39:12,375 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:a864f9f6) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
>>>>>>>>
>>>>>>>> 2022-10-17 13:40:13,276 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:445216f3) Trying to fetch storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297 from libvirt
>>>>>>>>
>>>>>>>> 2022-10-17 13:40:13,289 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:445216f3) Asking libvirt to refresh storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297
>>>>>>>>
>>>>>>>> 2022-10-17 13:40:13,378 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:445216f3) Trying to fetch storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89 from libvirt
>>>>>>>>
>>>>>>>> 2022-10-17 13:40:13,393 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:445216f3) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
>>>>>>>>
>>>>>>>> 2022-10-17 13:41:13,950 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:87ecfee0) Trying to fetch storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297 from libvirt
>>>>>>>>
>>>>>>>> 2022-10-17 13:41:13,964 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:87ecfee0) Asking libvirt to refresh storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297
>>>>>>>>
>>>>>>>> 2022-10-17 13:41:14,054 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:87ecfee0) Trying to fetch storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89 from libvirt
>>>>>>>>
>>>>>>>> 2022-10-17 13:41:14,067 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:87ecfee0) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
>>>>>>>>
>>>>>>>> 2022-10-17 13:41:57,753 WARN [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-3:null) (logid:d91c35e5) Expected 1 answers while executing SetMonitorServiceCommand but received 3
>>>>>>>>
>>>>>>>> 2022-10-17 13:41:57,938 INFO [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:null) (logid:1549c9a1) Fetching health check result for SEVERIPADDRESS and executing fresh checks: false
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 10/17/22 14:29, Nicolas Vazquez wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Hi Groete,
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Can you share the CloudStack agent logs from the KVM host?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Nicolas Vazquez
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> From: Granwille Strauss<gr...@namhost.com.INVALID> <ma...@namhost.com.INVALID> <gr...@namhost.com.INVALID> <ma...@namhost.com.INVALID> <ma...@namhost.com.INVALID> <gr...@namhost.com.INVALID> <ma...@namhost.com.INVALID> <ma...@namhost.com.INVALID> <gr...@namhost.com.INVALID> <ma...@namhost.com.INVALID> <ma...@namhost.com.INVALID> <gr...@namhost.com.INVALID> <ma...@namhost.com.INVALID>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Date: Monday, 17 October 2022 at 08:56
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> To:users@cloudstack.apache.org<ma...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Subject: Re: Recurring Snaps Failing
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Hi
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> I ran a few more tests and can now finally make sense of the failure:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> 2022-10-17 13:39:08,556 DEBUG [o.a.c.s.d.d.CloudStackPrimaryDataStoreDriverImpl] (Work-Job-Executor-1:ctx-93cb161b job-806/job-807 ctx-3f9ccf45) (logid:8fd10b20) Failed to take snapshot: 23
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> 2022-10-17 13:39:08,558 DEBUG [o.a.c.s.s.SnapshotServiceImpl] (Work-Job-Executor-1:ctx-93cb161b job-806/job-807 ctx-3f9ccf45) (logid:8fd10b20) create snapshot Test001 failed: com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:1, com.cloud.exception.OperationTimedoutException: Commands 7160441932542378205 to Host 1 timed out after 3600
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Seems to be a time out error. But there's no firewalls on the KVM host and NFS server to prevent connections and it worked fine before and randomly just stopped.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> So I still have no idea where to solve this as all other necessary settings are in place. Any ideas?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 10/17/22 08:51, Granwille Strauss wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Hi Guys
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> I enabled recurring snap shots for my Linux VMs and they all seem to be failing with the following error:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> root@hostname ~ $ grep error /var/log/cloudstack/management/management-server.log
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> 2022-10-17 03:01:29,474 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-27:ctx-99d7d702 job-796 ctx-f26a81fe) (logid:f9a61b00) Failed to create snapshot due to an internal error creating snapshot for volume 635fd5b5-bf14-4cf4-979c-be8a649d5189
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> 2022-10-17 03:01:29,476 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-27:ctx-99d7d702 job-796) (logid:f9a61b00) Complete async job-796, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 635fd5b5-bf14-4cf4-979c-be8a649d5189"}
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> 2022-10-17 03:01:29,478 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-26:ctx-c7865967 job-794 ctx-0700afb0) (logid:5541113c) Failed to create snapshot due to an internal error creating snapshot for volume 70ffef79-3aad-43ab-93d1-153fcd34ad15
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> 2022-10-17 03:01:29,480 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-26:ctx-c7865967 job-794) (logid:5541113c) Complete async job-794, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 70ffef79-3aad-43ab-93d1-153fcd34ad15"}
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> All I am getting out of this is internal error and nothing else. I have checked other logs and I am not quite sure where else to look.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Any assistance in this regard will be appreciated.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Regards / Groete
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Granwille Strauss // Senior Systems Admin
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> e:granwille@namhost.com<ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> m: +264 81 323 1260<tel:+264813231260> <tel:+264813231260><tel:+264813231260> <tel:+264813231260><tel:+264813231260> <tel:+264813231260><tel:+264813231260> <tel:+264813231260>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> w:www.namhost.com <http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com> <http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com> <http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com> <http://www.namhost.com> <http://www.namhost.com><https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Namhost Internet Services (Pty) Ltd,
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g> <https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Regards / Groete
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Granwille Strauss // Senior Systems Admin
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> e:granwille@namhost.com<ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> m: +264 81 323 1260<tel:+264813231260> <tel:+264813231260><tel:+264813231260> <tel:+264813231260><tel:+264813231260> <tel:+264813231260><tel:+264813231260> <tel:+264813231260>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> w:www.namhost.com <http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com> <http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com> <http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com> <http://www.namhost.com> <http://www.namhost.com><https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Namhost Internet Services (Pty) Ltd,
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g> <https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Regards / Groete
>>>>>>>>
>>>>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/>
>>>>>>>>
>>>>>>>> Granwille Strauss // Senior Systems Admin
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> e:granwille@namhost.com<ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com>
>>>>>>>>
>>>>>>>> m: +264 81 323 1260<tel:+264813231260> <tel:+264813231260><tel:+264813231260> <tel:+264813231260>
>>>>>>>>
>>>>>>>> w:www.namhost.com <http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com> <http://www.namhost.com> <http://www.namhost.com><https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>>>>>
>>>>>>>> Namhost Internet Services (Pty) Ltd,
>>>>>>>>
>>>>>>>>
>>>>>>>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g> <https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>>>>>>
>>>>>>>>
>>>>>>>> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Regards / Groete
>>>>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/>
>>>>>>>> Granwille Strauss // Senior Systems Admin
>>>>>>>>
>>>>>>>> e:granwille@namhost.com<ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com>
>>>>>>>> m: +264 81 323 1260<tel:+264813231260> <tel:+264813231260>
>>>>>>>> w:www.namhost.com <http://www.namhost.com><https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/>
>>>>>>>>
>>>>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>>>>>
>>>>>>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>>>>> Namhost Internet Services (Pty) Ltd,
>>>>>>>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g> <https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>>>>>>>
>>>>>>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>>>>>>
>>>>>>>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>>>>>> --
>>>>>>>>
>>>>>>>> Regards / Groete
>>>>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/>
>>>>>>>> Granwille Strauss // Senior Systems Admin
>>>>>>>>
>>>>>>>> e:granwille@namhost.com<ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com>
>>>>>>>> m: +264 81 323 1260<tel:+264813231260> <tel:+264813231260>
>>>>>>>> w:www.namhost.com <http://www.namhost.com><https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/>
>>>>>>>>
>>>>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>>>>>
>>>>>>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>>>>> Namhost Internet Services (Pty) Ltd,
>>>>>>>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g> <https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>>>>>>>
>>>>>>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and s
>>>>>>>>
>>>>>>>> --
>>>>>>>> Regards / Groete
>>>>>>>>
>>>>>>>> <https://www.namhost.com> <https://www.namhost.com> Granwille Strauss // Senior Systems Admin
>>>>>>>>
>>>>>>>> *e:*granwille@namhost.com
>>>>>>>> *m:* +264 81 323 1260 <+264813231260>
>>>>>>>> *w:*www.namhost.com <http://www.namhost.com>
>>>>>>>>
>>>>>>>> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost>
>>>>>>>> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/>
>>>>>>>> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos>
>>>>>>>> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>>>>>
>>>>>>>>
>>>>>>>> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>>>>>
>>>>>>>> Namhost Internet Services (Pty) Ltd,
>>>>>>>>
>>>>>>>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g> <https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> The content of this message is confidential. If you have received it by
>>>>>>>> mistake, please inform us by email reply and then delete the message. It is
>>>>>>>> forbidden to copy, forward, or in any way reveal the contents of this
>>>>>>>> message to anyone without our explicit consent. The integrity and security
>>>>>>>> of this email cannot be guaranteed over the Internet. Therefore, the sender
>>>>>>>> will not be held liable for any damage caused by the message. For our full
>>>>>>>> privacy policy and disclaimers, please go tohttps://www.namhost.com/
>>>>>>>> privacy-policy
>>>>>>>>
>>>>>>>> [image: Powered by AdSigner]
>>>>>>>> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>>>>>>
>>>>>> --
>>>>>> Regards / Groete
>>>>>>
>>>>>> <https://www.namhost.com> Granwille Strauss // Senior
>>>>>> Systems Admin
>>>>>>
>>>>>> *e:* granwille@namhost.com
>>>>>> *m:* +264 81 323 1260 <tel:+264813231260>
>>>>>> *w:* www.namhost.com <https://www.namhost.com/>
>>>>>>
>>>>>> <https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>>>
>>>>>> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>>>
>>>>>> Namhost Internet Services (Pty) Ltd,
>>>>>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>>>>>
>>>>>>
>>>>>> The content of this message is confidential. If you have
>>>>>> received it by mistake, please inform us by email reply
>>>>>> and then delete the message. It is forbidden to copy,
>>>>>> forward, or in any way reveal the contents of this
>>>>>> message to anyone without our explicit consent. The
>>>>>> integrity and security of this email cannot be guaranteed
>>>>>> over the Internet. Therefore, the sender will not be held
>>>>>> liable for any damage caused by the message. For our full
>>>>>> privacy policy and disclaimers, please go to
>>>>>> https://www.namhost.com/privacy-policy
>>>>>>
>>>>>> Powered by AdSigner
>>>>>> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>>> --
>>>>> Regards / Groete
>>>>>
>>>>> <https://www.namhost.com> Granwille Strauss // Senior
>>>>> Systems Admin
>>>>>
>>>>> *e:* granwille@namhost.com
>>>>> *m:* +264 81 323 1260 <tel:+264813231260>
>>>>> *w:* www.namhost.com <https://www.namhost.com/>
>>>>>
>>>>> <https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>>
>>>>> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>>
>>>>> Namhost Internet Services (Pty) Ltd,
>>>>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>>>>
>>>>>
>>>>> The content of this message is confidential. If you have
>>>>> received it by mistake, please inform us by email reply
>>>>> and then delete the message. It is forbidden to copy,
>>>>> forward, or in any way reveal the contents of this message
>>>>> to anyone without our explicit consent. The integrity and
>>>>> security of this email cannot be guaranteed over the
>>>>> Internet. Therefore, the sender will not be held liable
>>>>> for any damage caused by the message. For our full privacy
>>>>> policy and disclaimers, please go to
>>>>> https://www.namhost.com/privacy-policy
>>>>>
>>>>> Powered by AdSigner
>>>>> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>> --
>>>> Regards / Groete
>>>>
>>>> <https://www.namhost.com> Granwille Strauss // Senior
>>>> Systems Admin
>>>>
>>>> *e:* granwille@namhost.com
>>>> *m:* +264 81 323 1260 <tel:+264813231260>
>>>> *w:* www.namhost.com <https://www.namhost.com/>
>>>>
>>>> <https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>
>>>> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>
>>>> Namhost Internet Services (Pty) Ltd,
>>>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>>>
>>>>
>>>> The content of this message is confidential. If you have
>>>> received it by mistake, please inform us by email reply and
>>>> then delete the message. It is forbidden to copy, forward,
>>>> or in any way reveal the contents of this message to anyone
>>>> without our explicit consent. The integrity and security of
>>>> this email cannot be guaranteed over the Internet.
>>>> Therefore, the sender will not be held liable for any
>>>> damage caused by the message. For our full privacy policy
>>>> and disclaimers, please go to
>>>> https://www.namhost.com/privacy-policy
>>>>
>>>> Powered by AdSigner
>>>> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>> --
>>> Regards / Groete
>>>
>>> <https://www.namhost.com> Granwille Strauss // Senior
>>> Systems Admin
>>>
>>> *e:* granwille@namhost.com
>>> *m:* +264 81 323 1260 <tel:+264813231260>
>>> *w:* www.namhost.com <https://www.namhost.com/>
>>>
>>> <https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>
>>> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>
>>> Namhost Internet Services (Pty) Ltd,
>>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>>
>>>
>>> The content of this message is confidential. If you have
>>> received it by mistake, please inform us by email reply and
>>> then delete the message. It is forbidden to copy, forward,
>>> or in any way reveal the contents of this message to anyone
>>> without our explicit consent. The integrity and security of
>>> this email cannot be guaranteed over the Internet.
>>> Therefore, the sender will not be held liable for any damage
>>> caused by the message. For our full privacy policy and
>>> disclaimers, please go to
>>> https://www.namhost.com/privacy-policy
>>>
>>> Powered by AdSigner
>>> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>> --
>> Regards / Groete
>>
>> <https://www.namhost.com> Granwille Strauss // Senior
>> Systems Admin
>>
>> *e:* granwille@namhost.com
>> *m:* +264 81 323 1260 <tel:+264813231260>
>> *w:* www.namhost.com <https://www.namhost.com/>
>>
>> <https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>
>> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>
>> Namhost Internet Services (Pty) Ltd,
>>
>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>
>>
>>
>> The content of this message is confidential. If you have
>> received it by mistake, please inform us by email reply and
>> then delete the message. It is forbidden to copy, forward, or
>> in any way reveal the contents of this message to anyone
>> without our explicit consent. The integrity and security of
>> this email cannot be guaranteed over the Internet. Therefore,
>> the sender will not be held liable for any damage caused by
>> the message. For our full privacy policy and disclaimers,
>> please go to https://www.namhost.com/privacy-policy
>>
>> Powered by AdSigner
>> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>
> --
> Regards / Groete
>
> <https://www.namhost.com> Granwille Strauss // Senior Systems Admin
>
> *e:* granwille@namhost.com
> *m:* +264 81 323 1260 <tel:+264813231260>
> *w:* www.namhost.com <https://www.namhost.com/>
>
> <https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>
> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>
> Namhost Internet Services (Pty) Ltd,
>
> 24 Black Eagle Rd, Hermanus, 7210, RSA
>
>
>
> The content of this message is confidential. If you have received
> it by mistake, please inform us by email reply and then delete the
> message. It is forbidden to copy, forward, or in any way reveal
> the contents of this message to anyone without our explicit
> consent. The integrity and security of this email cannot be
> guaranteed over the Internet. Therefore, the sender will not be
> held liable for any damage caused by the message. For our full
> privacy policy and disclaimers, please go to
> https://www.namhost.com/privacy-policy
>
> Powered by AdSigner
> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>
--
Regards / Groete
<https://www.namhost.com> Granwille Strauss // Senior Systems Admin
*e:* granwille@namhost.com
*m:* +264 81 323 1260 <tel:+264813231260>
*w:* www.namhost.com <https://www.namhost.com/>
<https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
Namhost Internet Services (Pty) Ltd,
24 Black Eagle Rd, Hermanus, 7210, RSA
The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It
is forbidden to copy, forward, or in any way reveal the contents of this
message to anyone without our explicit consent. The integrity and
security of this email cannot be guaranteed over the Internet.
Therefore, the sender will not be held liable for any damage caused by
the message. For our full privacy policy and disclaimers, please go to
https://www.namhost.com/privacy-policy
Powered by AdSigner
<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
Re: Recurring Snaps Failing
Posted by Granwille Strauss <gr...@namhost.com.INVALID>.
Hi Wei
Thank you for the update. Thus far all the snapshot tests I took were
volume snaps and majority were when they are online and end up failing.
However, there's one VM where it seems to work just fine, whether its
online of offline and I am not sure what's different from this VM as all
other VM use the same Almalinux template. Its just this one VM where it
works as expected.
In any case, I did a VM snapshot now with memory on my Ubuntu vm while
it was online and the VM Snap was successful, no issues there. I ran a
volume snap right after on the same ubuntu vm and it failed again as
before, see attached screenshot. So VM snapshots seem to work fine, this
tells me the issue is most likely when its converted to QCOW2.
In any case, please see answers to your questions below:
1. I am not finding any .qcow2 images on the KVM server to list the
QCOW2 size, or am I missing something?
> root@HOSTNAME ~ $ find / -type f -iname '*.qcow2'
> root@HOSTNAME ~ $
As requested, here's a snapshot list of the current vm I am testing on.
Please let me know if you needed something specific?
> root@HOSTNAME ~ $ virsh snapshot-list i-2-19-VM
> Name Creation Time State
> -----------------------------------------------------------------------------------
> 0843e6b1-cb12-495c-8d5f-35c9a3338db7 2022-10-17 18:41:08 +0200
> disk-snapshot
> 2e77acb7-1be4-4417-858e-a1d0890f44f0 2022-10-17 19:57:03 +0200
> disk-snapshot
> 68476e2a-aec8-4637-9c4b-2dc49d79bd82 2022-10-18 09:44:21 +0200
> disk-snapshot
> c110058b-4661-42c0-b648-a54f4ba4beb7 2022-10-17 22:34:21 +0200
> disk-snapshot
> i-2-19-VM_VS_20221018072924 2022-10-18 09:29:25 +0200
> running
2. Attached is a txt file from yesterdays agent.log. This is all entries
that happened between CopyCommand and CopyCmdAnswer. There are zero
errors, as you will see.
3. The 'backup.snapshot.wait' setting is currently at 21600 seconds = 6
hours, should this still be increased? Also, not sure if
'copy.volume.wait' is relevant, but this is currently at 10800 seconds.
On 10/18/22 09:30, Wei ZHOU wrote:
> Hi Granwille,
>
> The process of volume snapshot on running vms and stop vms are
> different. For running vms, cloudstack will first take a vm snapshot
> and convert the vm snapshot into a qcow2 image.
> It is normal that snapshotting a running vm takes longer time than a
> stopped vm, because
> (1) there is an extra step to take a vm snapshot. Depending on the
> memory size, the step might take a few minutes.
> (3) vm snapshot is saved in QCOW2 image (it is called Internal
> snapshots). When taking a vm snapshot, the QCOW2 image will enlarge.
> The image will not shrink when the vm snapshot is removed.
> If you have taken vm snapshot (or volume snapshot on running vms)
> multiple times, The QCOW2 image might be very big. The whole process
> (including vm snapshotting and qemu-img convert) might exceed the
> timeout limit (3600 seconds = 1 hour).
>
> can you please
> (1) check the QCOW2 image size and list the snapshot of QCOW2 image
> (qemu-img info), list vm snapshot of the vm (virsh snapshot-list)
> (2) check the logs between CopyCommand and CopyCmdAnswer in agent.log
> (3) increase backup.snapshot.wait in global configuration and retry
> ("wait" in "CopyCommand" should be changed accordingly) ?
>
>
> -Wei
>
>
>
> On Tue, 18 Oct 2022 at 00:39, Granwille Strauss
> <gr...@namhost.com> wrote:
>
> Hi Guys
>
> I tested another Linux VM Almalinux VM, turned off the VM and made
> a successful snap. I turned it back online again and run another
> successful snap again. Now I run a snap on my Ubuntu VM and it
> failed. So just to be sure it wasn't just that one specific
> Almalinux VM that was working, I ran another snap on another
> Almalinux VM while it was online and too also failed. So I am not
> sure why recurring snaps is only working for this one VM and not
> the rest.
>
> Should I log this on GITHUB?
>
> On 10/17/22 21:55, Granwille Strauss wrote:
>>
>> Sorry, seems my attachments were too big, you can download via
>> Wetransfer: https://we.tl/t-9hUekbj86S
>>
>> And yes, I have the kvm.snapshot.enabled enabled in global
>> settings too, for live snapping.
>>
>> On 10/17/22 21:51, Granwille Strauss wrote:
>>>
>>> So it seems when VM is powered off, snap is successful. See
>>> attached logs for the successful snap, when VM is offline.
>>>
>>> On 10/17/22 21:38, Granwille Strauss wrote:
>>>>
>>>> Here's the last 10 minutes before the error is triggered in the
>>>> logs, this was the test run before switching off VM. That test
>>>> is still running and will provide its logs shortly, however, I
>>>> am confident, it will show same results.
>>>>
>>>> On 10/17/22 21:22, Granwille Strauss wrote:
>>>>>
>>>>> Hi Wei
>>>>>
>>>>> Please see attached. Please keep in mind, this worked 100% a
>>>>> few days ago when I tested on VM recurring snaps and it was
>>>>> perfect then. So I enabled recurring snaps for all 10 linux
>>>>> VMs I host. And I set them all to daily at 2AM.
>>>>> They failed after the first day. So I figured maybe too many
>>>>> concurrent jobs running at the same time.
>>>>>
>>>>> So I split them all to once weekly at different days at
>>>>> different hours. And still it failed. And now when doing it
>>>>> manually it fails too. But for some reason the snap ends up in
>>>>> /primary location is just returns an error for some reason.
>>>>>
>>>>> On 10/17/22 21:13, Wei ZHOU wrote:
>>>>>> Hi,
>>>>>>
>>>>>> What is the kvm host os and version ?
>>>>>>
>>>>>> -Wei
>>>>>>
>>>>>> On Monday, 17 October 2022, Granwille Strauss<gr...@namhost.com.invalid> <ma...@namhost.com.invalid>
>>>>>> wrote:
>>>>>>
>>>>>>> The error seems to happen exactly at 1 hour intervals. All of them have
>>>>>>> failed exactly at one hour. Also, these are all volume snapshots that I am
>>>>>>> taking, so the logs I sent you so far are all in this regard. They are not
>>>>>>> VM snaphots, nor are they powered off. I am running a new volume snap shot
>>>>>>> again, this time VM powered off lets see.
>>>>>>>
>>>>>>> I have also rebooted my entire KVM server and my management server. So far
>>>>>>> zero luck.
>>>>>>> On 10/17/22 20:26, Nicolas Vazquez wrote:
>>>>>>>
>>>>>>> Can you try taking a manual volume snapshot and share both mgmt and agent logs please?
>>>>>>>
>>>>>>> Regards,
>>>>>>> Nicolas Vazquez
>>>>>>>
>>>>>>>
>>>>>>> From: Granwille Strauss<gr...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com>
>>>>>>> Date: Monday, 17 October 2022 at 13:15
>>>>>>> To:users@cloudstack.apache.org <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org>
>>>>>>> Cc: Nicolas Vazquez<Ni...@shapeblue.com> <ma...@shapeblue.com> <Ni...@shapeblue.com> <ma...@shapeblue.com>
>>>>>>> Subject: Re: Recurring Snaps Failing
>>>>>>>
>>>>>>> Hi
>>>>>>>
>>>>>>> I also see another issue with snaps, I deleted all the failed snaps via the cloud management UI. But to my surprise, they're all still at /mnt/primary/snapshots/ see attached screenshot. Something odd is going on with snapshots.
>>>>>>> On 10/17/22 17:35, Granwille Strauss wrote:
>>>>>>>
>>>>>>> Hi
>>>>>>>
>>>>>>> I am not 100% if I enabled debug correctly, I followed thishttps://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+agent+debug and ran
>>>>>>> sed -i 's/INFO/DEBUG/g' /etc/cloudstack/agent/log4j-cloud.xml
>>>>>>>
>>>>>>> Attached are the two log files. Same error. I reboot SSVM & vRouter prior no luck.
>>>>>>> On 10/17/22 16:00, Granwille Strauss wrote:
>>>>>>>
>>>>>>> Hi
>>>>>>>
>>>>>>> No KVM is on its own server node and management on a different server node. However, management is is primary and secondary storage via NFS. I will enable DEBUG and re-run manual snap to replicate the error.
>>>>>>> On 10/17/22 15:42, Nicolas Vazquez wrote:
>>>>>>> Thanks, are the KVM agent and management server running on the same host? Can you increase the log level on the KVM agent logs to DEBUG and retry?
>>>>>>>
>>>>>>> Regards,
>>>>>>> Nicolas Vazquez
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> From: Granwille Strauss<gr...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com>
>>>>>>> Date: Monday, 17 October 2022 at 10:04
>>>>>>> To:users@cloudstack.apache.org<ma...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org>
>>>>>>> Cc: Nicolas Vazquez<Ni...@shapeblue.com> <ma...@shapeblue.com> <Ni...@shapeblue.com> <ma...@shapeblue.com> <ma...@shapeblue.com> <Ni...@shapeblue.com> <ma...@shapeblue.com>
>>>>>>> Subject: Re: Recurring Snaps Failing
>>>>>>>
>>>>>>> Please find attached TXT
>>>>>>> On 10/17/22 14:54, Nicolas Vazquez wrote:
>>>>>>>
>>>>>>> Ok, seems like the command is not reaching the KVM host, can you share the full management server logs for job-796?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Nicolas Vazquez
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> From: Granwille Strauss<gr...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com>
>>>>>>>
>>>>>>> Date: Monday, 17 October 2022 at 09:32
>>>>>>>
>>>>>>> To:users@cloudstack.apache.org<ma...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org>
>>>>>>>
>>>>>>> Cc: Nicolas Vazquez<Ni...@shapeblue.com> <ma...@shapeblue.com> <Ni...@shapeblue.com> <ma...@shapeblue.com> <ma...@shapeblue.com> <Ni...@shapeblue.com> <ma...@shapeblue.com>
>>>>>>>
>>>>>>> Subject: Re: Recurring Snaps Failing
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Hi
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> This is all agent threw during the snap process nothing else:
>>>>>>>
>>>>>>> 2022-10-17 13:39:12,375 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:a864f9f6) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
>>>>>>>
>>>>>>> 2022-10-17 13:40:13,276 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:445216f3) Trying to fetch storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297 from libvirt
>>>>>>>
>>>>>>> 2022-10-17 13:40:13,289 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:445216f3) Asking libvirt to refresh storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297
>>>>>>>
>>>>>>> 2022-10-17 13:40:13,378 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:445216f3) Trying to fetch storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89 from libvirt
>>>>>>>
>>>>>>> 2022-10-17 13:40:13,393 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:445216f3) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
>>>>>>>
>>>>>>> 2022-10-17 13:41:13,950 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:87ecfee0) Trying to fetch storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297 from libvirt
>>>>>>>
>>>>>>> 2022-10-17 13:41:13,964 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:87ecfee0) Asking libvirt to refresh storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297
>>>>>>>
>>>>>>> 2022-10-17 13:41:14,054 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:87ecfee0) Trying to fetch storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89 from libvirt
>>>>>>>
>>>>>>> 2022-10-17 13:41:14,067 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:87ecfee0) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
>>>>>>>
>>>>>>> 2022-10-17 13:41:57,753 WARN [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-3:null) (logid:d91c35e5) Expected 1 answers while executing SetMonitorServiceCommand but received 3
>>>>>>>
>>>>>>> 2022-10-17 13:41:57,938 INFO [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:null) (logid:1549c9a1) Fetching health check result for SEVERIPADDRESS and executing fresh checks: false
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 10/17/22 14:29, Nicolas Vazquez wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Hi Groete,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Can you share the CloudStack agent logs from the KVM host?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Nicolas Vazquez
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> From: Granwille Strauss<gr...@namhost.com.INVALID> <ma...@namhost.com.INVALID> <gr...@namhost.com.INVALID> <ma...@namhost.com.INVALID> <ma...@namhost.com.INVALID> <gr...@namhost.com.INVALID> <ma...@namhost.com.INVALID> <ma...@namhost.com.INVALID> <gr...@namhost.com.INVALID> <ma...@namhost.com.INVALID> <ma...@namhost.com.INVALID> <gr...@namhost.com.INVALID> <ma...@namhost.com.INVALID>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Date: Monday, 17 October 2022 at 08:56
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> To:users@cloudstack.apache.org<ma...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Subject: Re: Recurring Snaps Failing
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Hi
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I ran a few more tests and can now finally make sense of the failure:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2022-10-17 13:39:08,556 DEBUG [o.a.c.s.d.d.CloudStackPrimaryDataStoreDriverImpl] (Work-Job-Executor-1:ctx-93cb161b job-806/job-807 ctx-3f9ccf45) (logid:8fd10b20) Failed to take snapshot: 23
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2022-10-17 13:39:08,558 DEBUG [o.a.c.s.s.SnapshotServiceImpl] (Work-Job-Executor-1:ctx-93cb161b job-806/job-807 ctx-3f9ccf45) (logid:8fd10b20) create snapshot Test001 failed: com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:1, com.cloud.exception.OperationTimedoutException: Commands 7160441932542378205 to Host 1 timed out after 3600
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Seems to be a time out error. But there's no firewalls on the KVM host and NFS server to prevent connections and it worked fine before and randomly just stopped.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> So I still have no idea where to solve this as all other necessary settings are in place. Any ideas?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 10/17/22 08:51, Granwille Strauss wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Hi Guys
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I enabled recurring snap shots for my Linux VMs and they all seem to be failing with the following error:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> root@hostname ~ $ grep error /var/log/cloudstack/management/management-server.log
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2022-10-17 03:01:29,474 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-27:ctx-99d7d702 job-796 ctx-f26a81fe) (logid:f9a61b00) Failed to create snapshot due to an internal error creating snapshot for volume 635fd5b5-bf14-4cf4-979c-be8a649d5189
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2022-10-17 03:01:29,476 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-27:ctx-99d7d702 job-796) (logid:f9a61b00) Complete async job-796, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 635fd5b5-bf14-4cf4-979c-be8a649d5189"}
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2022-10-17 03:01:29,478 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-26:ctx-c7865967 job-794 ctx-0700afb0) (logid:5541113c) Failed to create snapshot due to an internal error creating snapshot for volume 70ffef79-3aad-43ab-93d1-153fcd34ad15
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2022-10-17 03:01:29,480 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-26:ctx-c7865967 job-794) (logid:5541113c) Complete async job-794, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 70ffef79-3aad-43ab-93d1-153fcd34ad15"}
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> All I am getting out of this is internal error and nothing else. I have checked other logs and I am not quite sure where else to look.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Any assistance in this regard will be appreciated.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Regards / Groete
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Granwille Strauss // Senior Systems Admin
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> e:granwille@namhost.com<ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> m: +264 81 323 1260<tel:+264813231260> <tel:+264813231260><tel:+264813231260> <tel:+264813231260><tel:+264813231260> <tel:+264813231260><tel:+264813231260> <tel:+264813231260>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> w:www.namhost.com <http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com> <http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com> <http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com> <http://www.namhost.com> <http://www.namhost.com><https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Namhost Internet Services (Pty) Ltd,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g> <https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Regards / Groete
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Granwille Strauss // Senior Systems Admin
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> e:granwille@namhost.com<ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> m: +264 81 323 1260<tel:+264813231260> <tel:+264813231260><tel:+264813231260> <tel:+264813231260><tel:+264813231260> <tel:+264813231260><tel:+264813231260> <tel:+264813231260>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> w:www.namhost.com <http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com> <http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com> <http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com> <http://www.namhost.com> <http://www.namhost.com><https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Namhost Internet Services (Pty) Ltd,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g> <https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Regards / Groete
>>>>>>>
>>>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/>
>>>>>>>
>>>>>>> Granwille Strauss // Senior Systems Admin
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> e:granwille@namhost.com<ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com>
>>>>>>>
>>>>>>> m: +264 81 323 1260<tel:+264813231260> <tel:+264813231260><tel:+264813231260> <tel:+264813231260>
>>>>>>>
>>>>>>> w:www.namhost.com <http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com> <http://www.namhost.com> <http://www.namhost.com><https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>>>>
>>>>>>> Namhost Internet Services (Pty) Ltd,
>>>>>>>
>>>>>>>
>>>>>>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g> <https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>>>>>
>>>>>>>
>>>>>>> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Regards / Groete
>>>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/>
>>>>>>> Granwille Strauss // Senior Systems Admin
>>>>>>>
>>>>>>> e:granwille@namhost.com<ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com>
>>>>>>> m: +264 81 323 1260<tel:+264813231260> <tel:+264813231260>
>>>>>>> w:www.namhost.com <http://www.namhost.com><https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/>
>>>>>>>
>>>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>>>>
>>>>>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>>>> Namhost Internet Services (Pty) Ltd,
>>>>>>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g> <https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>>>>>>
>>>>>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>>>>>
>>>>>>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>>>>> --
>>>>>>>
>>>>>>> Regards / Groete
>>>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/>
>>>>>>> Granwille Strauss // Senior Systems Admin
>>>>>>>
>>>>>>> e:granwille@namhost.com<ma...@namhost.com> <ma...@namhost.com> <gr...@namhost.com> <ma...@namhost.com>
>>>>>>> m: +264 81 323 1260<tel:+264813231260> <tel:+264813231260>
>>>>>>> w:www.namhost.com <http://www.namhost.com><https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/> <https://www.namhost.com/>
>>>>>>>
>>>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>>>>
>>>>>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>>>> Namhost Internet Services (Pty) Ltd,
>>>>>>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g> <https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>>>>>>
>>>>>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and s
>>>>>>>
>>>>>>> --
>>>>>>> Regards / Groete
>>>>>>>
>>>>>>> <https://www.namhost.com> <https://www.namhost.com> Granwille Strauss // Senior Systems Admin
>>>>>>>
>>>>>>> *e:*granwille@namhost.com
>>>>>>> *m:* +264 81 323 1260 <+264813231260>
>>>>>>> *w:*www.namhost.com <http://www.namhost.com>
>>>>>>>
>>>>>>> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost> <https://twitter.com/namhost> <https://twitter.com/namhost>
>>>>>>> <https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/>
>>>>>>> <https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos>
>>>>>>> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>>>>
>>>>>>>
>>>>>>> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>>>>
>>>>>>> Namhost Internet Services (Pty) Ltd,
>>>>>>>
>>>>>>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g> <https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> The content of this message is confidential. If you have received it by
>>>>>>> mistake, please inform us by email reply and then delete the message. It is
>>>>>>> forbidden to copy, forward, or in any way reveal the contents of this
>>>>>>> message to anyone without our explicit consent. The integrity and security
>>>>>>> of this email cannot be guaranteed over the Internet. Therefore, the sender
>>>>>>> will not be held liable for any damage caused by the message. For our full
>>>>>>> privacy policy and disclaimers, please go tohttps://www.namhost.com/
>>>>>>> privacy-policy
>>>>>>>
>>>>>>> [image: Powered by AdSigner]
>>>>>>> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>>>>>
>>>>> --
>>>>> Regards / Groete
>>>>>
>>>>> <https://www.namhost.com> Granwille Strauss // Senior
>>>>> Systems Admin
>>>>>
>>>>> *e:* granwille@namhost.com
>>>>> *m:* +264 81 323 1260 <tel:+264813231260>
>>>>> *w:* www.namhost.com <https://www.namhost.com/>
>>>>>
>>>>> <https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>>
>>>>> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>>
>>>>> Namhost Internet Services (Pty) Ltd,
>>>>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>>>>
>>>>>
>>>>> The content of this message is confidential. If you have
>>>>> received it by mistake, please inform us by email reply and
>>>>> then delete the message. It is forbidden to copy, forward, or
>>>>> in any way reveal the contents of this message to anyone
>>>>> without our explicit consent. The integrity and security of
>>>>> this email cannot be guaranteed over the Internet. Therefore,
>>>>> the sender will not be held liable for any damage caused by
>>>>> the message. For our full privacy policy and disclaimers,
>>>>> please go to https://www.namhost.com/privacy-policy
>>>>>
>>>>> Powered by AdSigner
>>>>> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>> --
>>>> Regards / Groete
>>>>
>>>> <https://www.namhost.com> Granwille Strauss // Senior Systems
>>>> Admin
>>>>
>>>> *e:* granwille@namhost.com
>>>> *m:* +264 81 323 1260 <tel:+264813231260>
>>>> *w:* www.namhost.com <https://www.namhost.com/>
>>>>
>>>> <https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>
>>>> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>
>>>> Namhost Internet Services (Pty) Ltd,
>>>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>>>
>>>>
>>>> The content of this message is confidential. If you have
>>>> received it by mistake, please inform us by email reply and
>>>> then delete the message. It is forbidden to copy, forward, or
>>>> in any way reveal the contents of this message to anyone
>>>> without our explicit consent. The integrity and security of
>>>> this email cannot be guaranteed over the Internet. Therefore,
>>>> the sender will not be held liable for any damage caused by the
>>>> message. For our full privacy policy and disclaimers, please go
>>>> to https://www.namhost.com/privacy-policy
>>>>
>>>> Powered by AdSigner
>>>> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>> --
>>> Regards / Groete
>>>
>>> <https://www.namhost.com> Granwille Strauss // Senior Systems
>>> Admin
>>>
>>> *e:* granwille@namhost.com
>>> *m:* +264 81 323 1260 <tel:+264813231260>
>>> *w:* www.namhost.com <https://www.namhost.com/>
>>>
>>> <https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>
>>> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>
>>> Namhost Internet Services (Pty) Ltd,
>>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>>
>>>
>>> The content of this message is confidential. If you have
>>> received it by mistake, please inform us by email reply and then
>>> delete the message. It is forbidden to copy, forward, or in any
>>> way reveal the contents of this message to anyone without our
>>> explicit consent. The integrity and security of this email
>>> cannot be guaranteed over the Internet. Therefore, the sender
>>> will not be held liable for any damage caused by the message.
>>> For our full privacy policy and disclaimers, please go to
>>> https://www.namhost.com/privacy-policy
>>>
>>> Powered by AdSigner
>>> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>> --
>> Regards / Groete
>>
>> <https://www.namhost.com> Granwille Strauss // Senior Systems Admin
>>
>> *e:* granwille@namhost.com
>> *m:* +264 81 323 1260 <tel:+264813231260>
>> *w:* www.namhost.com <https://www.namhost.com/>
>>
>> <https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>
>> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>
>> Namhost Internet Services (Pty) Ltd,
>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>
>>
>> The content of this message is confidential. If you have received
>> it by mistake, please inform us by email reply and then delete
>> the message. It is forbidden to copy, forward, or in any way
>> reveal the contents of this message to anyone without our
>> explicit consent. The integrity and security of this email cannot
>> be guaranteed over the Internet. Therefore, the sender will not
>> be held liable for any damage caused by the message. For our full
>> privacy policy and disclaimers, please go to
>> https://www.namhost.com/privacy-policy
>>
>> Powered by AdSigner
>> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
> --
> Regards / Groete
>
> <https://www.namhost.com> Granwille Strauss // Senior Systems Admin
>
> *e:* granwille@namhost.com
> *m:* +264 81 323 1260 <tel:+264813231260>
> *w:* www.namhost.com <https://www.namhost.com/>
>
> <https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>
> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>
> Namhost Internet Services (Pty) Ltd,
>
> 24 Black Eagle Rd, Hermanus, 7210, RSA
>
>
>
> The content of this message is confidential. If you have received
> it by mistake, please inform us by email reply and then delete the
> message. It is forbidden to copy, forward, or in any way reveal
> the contents of this message to anyone without our explicit
> consent. The integrity and security of this email cannot be
> guaranteed over the Internet. Therefore, the sender will not be
> held liable for any damage caused by the message. For our full
> privacy policy and disclaimers, please go to
> https://www.namhost.com/privacy-policy
>
> Powered by AdSigner
> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>
--
Regards / Groete
<https://www.namhost.com> Granwille Strauss // Senior Systems Admin
*e:* granwille@namhost.com
*m:* +264 81 323 1260 <tel:+264813231260>
*w:* www.namhost.com <https://www.namhost.com/>
<https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
Namhost Internet Services (Pty) Ltd,
24 Black Eagle Rd, Hermanus, 7210, RSA
The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It
is forbidden to copy, forward, or in any way reveal the contents of this
message to anyone without our explicit consent. The integrity and
security of this email cannot be guaranteed over the Internet.
Therefore, the sender will not be held liable for any damage caused by
the message. For our full privacy policy and disclaimers, please go to
https://www.namhost.com/privacy-policy
Powered by AdSigner
<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
Re: Recurring Snaps Failing
Posted by Granwille Strauss <gr...@namhost.com.INVALID>.
Hi Guys
I tested another Linux VM Almalinux VM, turned off the VM and made a
successful snap. I turned it back online again and run another
successful snap again. Now I run a snap on my Ubuntu VM and it failed.
So just to be sure it wasn't just that one specific Almalinux VM that
was working, I ran another snap on another Almalinux VM while it was
online and too also failed. So I am not sure why recurring snaps is only
working for this one VM and not the rest.
Should I log this on GITHUB?
On 10/17/22 21:55, Granwille Strauss wrote:
>
> Sorry, seems my attachments were too big, you can download via
> Wetransfer: https://we.tl/t-9hUekbj86S
>
> And yes, I have the kvm.snapshot.enabled enabled in global settings
> too, for live snapping.
>
> On 10/17/22 21:51, Granwille Strauss wrote:
>>
>> So it seems when VM is powered off, snap is successful. See attached
>> logs for the successful snap, when VM is offline.
>>
>> On 10/17/22 21:38, Granwille Strauss wrote:
>>>
>>> Here's the last 10 minutes before the error is triggered in the
>>> logs, this was the test run before switching off VM. That test is
>>> still running and will provide its logs shortly, however, I am
>>> confident, it will show same results.
>>>
>>> On 10/17/22 21:22, Granwille Strauss wrote:
>>>>
>>>> Hi Wei
>>>>
>>>> Please see attached. Please keep in mind, this worked 100% a few
>>>> days ago when I tested on VM recurring snaps and it was perfect
>>>> then. So I enabled recurring snaps for all 10 linux VMs I host. And
>>>> I set them all to daily at 2AM.
>>>> They failed after the first day. So I figured maybe too many
>>>> concurrent jobs running at the same time.
>>>>
>>>> So I split them all to once weekly at different days at different
>>>> hours. And still it failed. And now when doing it manually it fails
>>>> too. But for some reason the snap ends up in /primary location is
>>>> just returns an error for some reason.
>>>>
>>>> On 10/17/22 21:13, Wei ZHOU wrote:
>>>>> Hi,
>>>>>
>>>>> What is the kvm host os and version ?
>>>>>
>>>>> -Wei
>>>>>
>>>>> On Monday, 17 October 2022, Granwille Strauss<gr...@namhost.com.invalid>
>>>>> wrote:
>>>>>
>>>>>> The error seems to happen exactly at 1 hour intervals. All of them have
>>>>>> failed exactly at one hour. Also, these are all volume snapshots that I am
>>>>>> taking, so the logs I sent you so far are all in this regard. They are not
>>>>>> VM snaphots, nor are they powered off. I am running a new volume snap shot
>>>>>> again, this time VM powered off lets see.
>>>>>>
>>>>>> I have also rebooted my entire KVM server and my management server. So far
>>>>>> zero luck.
>>>>>> On 10/17/22 20:26, Nicolas Vazquez wrote:
>>>>>>
>>>>>> Can you try taking a manual volume snapshot and share both mgmt and agent logs please?
>>>>>>
>>>>>> Regards,
>>>>>> Nicolas Vazquez
>>>>>>
>>>>>>
>>>>>> From: Granwille Strauss<gr...@namhost.com> <gr...@namhost.com>
>>>>>> Date: Monday, 17 October 2022 at 13:15
>>>>>> To:users@cloudstack.apache.org <us...@cloudstack.apache.org> <us...@cloudstack.apache.org>
>>>>>> Cc: Nicolas Vazquez<Ni...@shapeblue.com> <Ni...@shapeblue.com>
>>>>>> Subject: Re: Recurring Snaps Failing
>>>>>>
>>>>>> Hi
>>>>>>
>>>>>> I also see another issue with snaps, I deleted all the failed snaps via the cloud management UI. But to my surprise, they're all still at /mnt/primary/snapshots/ see attached screenshot. Something odd is going on with snapshots.
>>>>>> On 10/17/22 17:35, Granwille Strauss wrote:
>>>>>>
>>>>>> Hi
>>>>>>
>>>>>> I am not 100% if I enabled debug correctly, I followed thishttps://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+agent+debug and ran
>>>>>> sed -i 's/INFO/DEBUG/g' /etc/cloudstack/agent/log4j-cloud.xml
>>>>>>
>>>>>> Attached are the two log files. Same error. I reboot SSVM & vRouter prior no luck.
>>>>>> On 10/17/22 16:00, Granwille Strauss wrote:
>>>>>>
>>>>>> Hi
>>>>>>
>>>>>> No KVM is on its own server node and management on a different server node. However, management is is primary and secondary storage via NFS. I will enable DEBUG and re-run manual snap to replicate the error.
>>>>>> On 10/17/22 15:42, Nicolas Vazquez wrote:
>>>>>> Thanks, are the KVM agent and management server running on the same host? Can you increase the log level on the KVM agent logs to DEBUG and retry?
>>>>>>
>>>>>> Regards,
>>>>>> Nicolas Vazquez
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> From: Granwille Strauss<gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com>
>>>>>> Date: Monday, 17 October 2022 at 10:04
>>>>>> To:users@cloudstack.apache.org<ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org>
>>>>>> Cc: Nicolas Vazquez<Ni...@shapeblue.com> <Ni...@shapeblue.com> <Ni...@shapeblue.com>
>>>>>> Subject: Re: Recurring Snaps Failing
>>>>>>
>>>>>> Please find attached TXT
>>>>>> On 10/17/22 14:54, Nicolas Vazquez wrote:
>>>>>>
>>>>>> Ok, seems like the command is not reaching the KVM host, can you share the full management server logs for job-796?
>>>>>>
>>>>>>
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Nicolas Vazquez
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> From: Granwille Strauss<gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com>
>>>>>>
>>>>>> Date: Monday, 17 October 2022 at 09:32
>>>>>>
>>>>>> To:users@cloudstack.apache.org<ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org>
>>>>>>
>>>>>> Cc: Nicolas Vazquez<Ni...@shapeblue.com> <Ni...@shapeblue.com> <Ni...@shapeblue.com>
>>>>>>
>>>>>> Subject: Re: Recurring Snaps Failing
>>>>>>
>>>>>>
>>>>>>
>>>>>> Hi
>>>>>>
>>>>>>
>>>>>>
>>>>>> This is all agent threw during the snap process nothing else:
>>>>>>
>>>>>> 2022-10-17 13:39:12,375 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:a864f9f6) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
>>>>>>
>>>>>> 2022-10-17 13:40:13,276 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:445216f3) Trying to fetch storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297 from libvirt
>>>>>>
>>>>>> 2022-10-17 13:40:13,289 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:445216f3) Asking libvirt to refresh storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297
>>>>>>
>>>>>> 2022-10-17 13:40:13,378 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:445216f3) Trying to fetch storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89 from libvirt
>>>>>>
>>>>>> 2022-10-17 13:40:13,393 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:445216f3) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
>>>>>>
>>>>>> 2022-10-17 13:41:13,950 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:87ecfee0) Trying to fetch storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297 from libvirt
>>>>>>
>>>>>> 2022-10-17 13:41:13,964 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:87ecfee0) Asking libvirt to refresh storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297
>>>>>>
>>>>>> 2022-10-17 13:41:14,054 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:87ecfee0) Trying to fetch storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89 from libvirt
>>>>>>
>>>>>> 2022-10-17 13:41:14,067 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:87ecfee0) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
>>>>>>
>>>>>> 2022-10-17 13:41:57,753 WARN [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-3:null) (logid:d91c35e5) Expected 1 answers while executing SetMonitorServiceCommand but received 3
>>>>>>
>>>>>> 2022-10-17 13:41:57,938 INFO [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:null) (logid:1549c9a1) Fetching health check result for SEVERIPADDRESS and executing fresh checks: false
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 10/17/22 14:29, Nicolas Vazquez wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> Hi Groete,
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Can you share the CloudStack agent logs from the KVM host?
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>>
>>>>>>
>>>>>> Nicolas Vazquez
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> From: Granwille Strauss<gr...@namhost.com.INVALID> <gr...@namhost.com.INVALID> <gr...@namhost.com.INVALID> <gr...@namhost.com.INVALID> <gr...@namhost.com.INVALID>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Date: Monday, 17 October 2022 at 08:56
>>>>>>
>>>>>>
>>>>>>
>>>>>> To:users@cloudstack.apache.org<ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Subject: Re: Recurring Snaps Failing
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Hi
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> I ran a few more tests and can now finally make sense of the failure:
>>>>>>
>>>>>>
>>>>>>
>>>>>> 2022-10-17 13:39:08,556 DEBUG [o.a.c.s.d.d.CloudStackPrimaryDataStoreDriverImpl] (Work-Job-Executor-1:ctx-93cb161b job-806/job-807 ctx-3f9ccf45) (logid:8fd10b20) Failed to take snapshot: 23
>>>>>>
>>>>>>
>>>>>>
>>>>>> 2022-10-17 13:39:08,558 DEBUG [o.a.c.s.s.SnapshotServiceImpl] (Work-Job-Executor-1:ctx-93cb161b job-806/job-807 ctx-3f9ccf45) (logid:8fd10b20) create snapshot Test001 failed: com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:1, com.cloud.exception.OperationTimedoutException: Commands 7160441932542378205 to Host 1 timed out after 3600
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Seems to be a time out error. But there's no firewalls on the KVM host and NFS server to prevent connections and it worked fine before and randomly just stopped.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> So I still have no idea where to solve this as all other necessary settings are in place. Any ideas?
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 10/17/22 08:51, Granwille Strauss wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Hi Guys
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> I enabled recurring snap shots for my Linux VMs and they all seem to be failing with the following error:
>>>>>>
>>>>>>
>>>>>>
>>>>>> root@hostname ~ $ grep error /var/log/cloudstack/management/management-server.log
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> 2022-10-17 03:01:29,474 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-27:ctx-99d7d702 job-796 ctx-f26a81fe) (logid:f9a61b00) Failed to create snapshot due to an internal error creating snapshot for volume 635fd5b5-bf14-4cf4-979c-be8a649d5189
>>>>>>
>>>>>>
>>>>>>
>>>>>> 2022-10-17 03:01:29,476 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-27:ctx-99d7d702 job-796) (logid:f9a61b00) Complete async job-796, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 635fd5b5-bf14-4cf4-979c-be8a649d5189"}
>>>>>>
>>>>>>
>>>>>>
>>>>>> 2022-10-17 03:01:29,478 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-26:ctx-c7865967 job-794 ctx-0700afb0) (logid:5541113c) Failed to create snapshot due to an internal error creating snapshot for volume 70ffef79-3aad-43ab-93d1-153fcd34ad15
>>>>>>
>>>>>>
>>>>>>
>>>>>> 2022-10-17 03:01:29,480 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-26:ctx-c7865967 job-794) (logid:5541113c) Complete async job-794, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 70ffef79-3aad-43ab-93d1-153fcd34ad15"}
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> All I am getting out of this is internal error and nothing else. I have checked other logs and I am not quite sure where else to look.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Any assistance in this regard will be appreciated.
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Regards / Groete
>>>>>>
>>>>>>
>>>>>>
>>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Granwille Strauss // Senior Systems Admin
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> e:granwille@namhost.com<ma...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com>
>>>>>>
>>>>>>
>>>>>>
>>>>>> m: +264 81 323 1260<tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260>
>>>>>>
>>>>>>
>>>>>>
>>>>>> w:www.namhost.com<http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Namhost Internet Services (Pty) Ltd,
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Regards / Groete
>>>>>>
>>>>>>
>>>>>>
>>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Granwille Strauss // Senior Systems Admin
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> e:granwille@namhost.com<ma...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com>
>>>>>>
>>>>>>
>>>>>>
>>>>>> m: +264 81 323 1260<tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260>
>>>>>>
>>>>>>
>>>>>>
>>>>>> w:www.namhost.com<http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Namhost Internet Services (Pty) Ltd,
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>>
>>>>>>
>>>>>> Regards / Groete
>>>>>>
>>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/>
>>>>>>
>>>>>> Granwille Strauss // Senior Systems Admin
>>>>>>
>>>>>>
>>>>>>
>>>>>> e:granwille@namhost.com<ma...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com>
>>>>>>
>>>>>> m: +264 81 323 1260<tel:+264813231260><tel:+264813231260>
>>>>>>
>>>>>> w:www.namhost.com<http://www.namhost.com> <http://www.namhost.com><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/>
>>>>>>
>>>>>>
>>>>>>
>>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>>>
>>>>>>
>>>>>>
>>>>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>>>
>>>>>> Namhost Internet Services (Pty) Ltd,
>>>>>>
>>>>>>
>>>>>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>>>>>
>>>>>>
>>>>>>
>>>>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>>>>
>>>>>>
>>>>>> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Regards / Groete
>>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/> <https://www.namhost.com/>
>>>>>> Granwille Strauss // Senior Systems Admin
>>>>>>
>>>>>> e:granwille@namhost.com<ma...@namhost.com> <gr...@namhost.com>
>>>>>> m: +264 81 323 1260<tel:+264813231260>
>>>>>> w:www.namhost.com<https://www.namhost.com/> <https://www.namhost.com/>
>>>>>>
>>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost> <https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost> <https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>>>
>>>>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>>> Namhost Internet Services (Pty) Ltd,
>>>>>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>>>>>
>>>>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>>>>
>>>>>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>>>> --
>>>>>>
>>>>>> Regards / Groete
>>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/> <https://www.namhost.com/>
>>>>>> Granwille Strauss // Senior Systems Admin
>>>>>>
>>>>>> e:granwille@namhost.com<ma...@namhost.com> <gr...@namhost.com>
>>>>>> m: +264 81 323 1260<tel:+264813231260>
>>>>>> w:www.namhost.com<https://www.namhost.com/> <https://www.namhost.com/>
>>>>>>
>>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost> <https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost> <https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>>>
>>>>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>>> Namhost Internet Services (Pty) Ltd,
>>>>>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>>>>>
>>>>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and s
>>>>>>
>>>>>> --
>>>>>> Regards / Groete
>>>>>>
>>>>>> <https://www.namhost.com> Granwille Strauss // Senior Systems Admin
>>>>>>
>>>>>> *e:*granwille@namhost.com
>>>>>> *m:* +264 81 323 1260 <+264813231260>
>>>>>> *w:*www.namhost.com
>>>>>>
>>>>>> <https://www.facebook.com/namhost> <https://twitter.com/namhost>
>>>>>> <https://www.instagram.com/namhostinternetservices/>
>>>>>> <https://www.linkedin.com/company/namhos>
>>>>>> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>>>
>>>>>>
>>>>>> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>>>
>>>>>> Namhost Internet Services (Pty) Ltd,
>>>>>>
>>>>>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>>>>>
>>>>>>
>>>>>>
>>>>>> The content of this message is confidential. If you have received it by
>>>>>> mistake, please inform us by email reply and then delete the message. It is
>>>>>> forbidden to copy, forward, or in any way reveal the contents of this
>>>>>> message to anyone without our explicit consent. The integrity and security
>>>>>> of this email cannot be guaranteed over the Internet. Therefore, the sender
>>>>>> will not be held liable for any damage caused by the message. For our full
>>>>>> privacy policy and disclaimers, please go tohttps://www.namhost.com/
>>>>>> privacy-policy
>>>>>>
>>>>>> [image: Powered by AdSigner]
>>>>>> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>>>>
>>>> --
>>>> Regards / Groete
>>>>
>>>> <https://www.namhost.com> Granwille Strauss // Senior Systems Admin
>>>>
>>>> *e:* granwille@namhost.com
>>>> *m:* +264 81 323 1260 <tel:+264813231260>
>>>> *w:* www.namhost.com <https://www.namhost.com/>
>>>>
>>>> <https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>
>>>> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>
>>>> Namhost Internet Services (Pty) Ltd,
>>>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>>>
>>>>
>>>> The content of this message is confidential. If you have received
>>>> it by mistake, please inform us by email reply and then delete the
>>>> message. It is forbidden to copy, forward, or in any way reveal the
>>>> contents of this message to anyone without our explicit consent.
>>>> The integrity and security of this email cannot be guaranteed over
>>>> the Internet. Therefore, the sender will not be held liable for any
>>>> damage caused by the message. For our full privacy policy and
>>>> disclaimers, please go to https://www.namhost.com/privacy-policy
>>>>
>>>> Powered by AdSigner
>>>> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>> --
>>> Regards / Groete
>>>
>>> <https://www.namhost.com> Granwille Strauss // Senior Systems Admin
>>>
>>> *e:* granwille@namhost.com
>>> *m:* +264 81 323 1260 <tel:+264813231260>
>>> *w:* www.namhost.com <https://www.namhost.com/>
>>>
>>> <https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>
>>> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>
>>> Namhost Internet Services (Pty) Ltd,
>>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>>
>>>
>>> The content of this message is confidential. If you have received it
>>> by mistake, please inform us by email reply and then delete the
>>> message. It is forbidden to copy, forward, or in any way reveal the
>>> contents of this message to anyone without our explicit consent. The
>>> integrity and security of this email cannot be guaranteed over the
>>> Internet. Therefore, the sender will not be held liable for any
>>> damage caused by the message. For our full privacy policy and
>>> disclaimers, please go to https://www.namhost.com/privacy-policy
>>>
>>> Powered by AdSigner
>>> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>> --
>> Regards / Groete
>>
>> <https://www.namhost.com> Granwille Strauss // Senior Systems Admin
>>
>> *e:* granwille@namhost.com
>> *m:* +264 81 323 1260 <tel:+264813231260>
>> *w:* www.namhost.com <https://www.namhost.com/>
>>
>> <https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>
>> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>
>> Namhost Internet Services (Pty) Ltd,
>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>
>>
>> The content of this message is confidential. If you have received it
>> by mistake, please inform us by email reply and then delete the
>> message. It is forbidden to copy, forward, or in any way reveal the
>> contents of this message to anyone without our explicit consent. The
>> integrity and security of this email cannot be guaranteed over the
>> Internet. Therefore, the sender will not be held liable for any
>> damage caused by the message. For our full privacy policy and
>> disclaimers, please go to https://www.namhost.com/privacy-policy
>>
>> Powered by AdSigner
>> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
> --
> Regards / Groete
>
> <https://www.namhost.com> Granwille Strauss // Senior Systems Admin
>
> *e:* granwille@namhost.com
> *m:* +264 81 323 1260 <tel:+264813231260>
> *w:* www.namhost.com <https://www.namhost.com/>
>
> <https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>
> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>
> Namhost Internet Services (Pty) Ltd,
> 24 Black Eagle Rd, Hermanus, 7210, RSA
>
>
> The content of this message is confidential. If you have received it
> by mistake, please inform us by email reply and then delete the
> message. It is forbidden to copy, forward, or in any way reveal the
> contents of this message to anyone without our explicit consent. The
> integrity and security of this email cannot be guaranteed over the
> Internet. Therefore, the sender will not be held liable for any damage
> caused by the message. For our full privacy policy and disclaimers,
> please go to https://www.namhost.com/privacy-policy
>
> Powered by AdSigner
> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
--
Regards / Groete
<https://www.namhost.com> Granwille Strauss // Senior Systems Admin
*e:* granwille@namhost.com
*m:* +264 81 323 1260 <tel:+264813231260>
*w:* www.namhost.com <https://www.namhost.com/>
<https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
Namhost Internet Services (Pty) Ltd,
24 Black Eagle Rd, Hermanus, 7210, RSA
The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It
is forbidden to copy, forward, or in any way reveal the contents of this
message to anyone without our explicit consent. The integrity and
security of this email cannot be guaranteed over the Internet.
Therefore, the sender will not be held liable for any damage caused by
the message. For our full privacy policy and disclaimers, please go to
https://www.namhost.com/privacy-policy
Powered by AdSigner
<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
Re: Recurring Snaps Failing
Posted by Granwille Strauss <gr...@namhost.com.INVALID>.
Sorry, seems my attachments were too big, you can download via
Wetransfer: https://we.tl/t-9hUekbj86S
And yes, I have the kvm.snapshot.enabled enabled in global settings too,
for live snapping.
On 10/17/22 21:51, Granwille Strauss wrote:
>
> So it seems when VM is powered off, snap is successful. See attached
> logs for the successful snap, when VM is offline.
>
> On 10/17/22 21:38, Granwille Strauss wrote:
>>
>> Here's the last 10 minutes before the error is triggered in the logs,
>> this was the test run before switching off VM. That test is still
>> running and will provide its logs shortly, however, I am confident,
>> it will show same results.
>>
>> On 10/17/22 21:22, Granwille Strauss wrote:
>>>
>>> Hi Wei
>>>
>>> Please see attached. Please keep in mind, this worked 100% a few
>>> days ago when I tested on VM recurring snaps and it was perfect
>>> then. So I enabled recurring snaps for all 10 linux VMs I host. And
>>> I set them all to daily at 2AM.
>>> They failed after the first day. So I figured maybe too many
>>> concurrent jobs running at the same time.
>>>
>>> So I split them all to once weekly at different days at different
>>> hours. And still it failed. And now when doing it manually it fails
>>> too. But for some reason the snap ends up in /primary location is
>>> just returns an error for some reason.
>>>
>>> On 10/17/22 21:13, Wei ZHOU wrote:
>>>> Hi,
>>>>
>>>> What is the kvm host os and version ?
>>>>
>>>> -Wei
>>>>
>>>> On Monday, 17 October 2022, Granwille Strauss<gr...@namhost.com.invalid>
>>>> wrote:
>>>>
>>>>> The error seems to happen exactly at 1 hour intervals. All of them have
>>>>> failed exactly at one hour. Also, these are all volume snapshots that I am
>>>>> taking, so the logs I sent you so far are all in this regard. They are not
>>>>> VM snaphots, nor are they powered off. I am running a new volume snap shot
>>>>> again, this time VM powered off lets see.
>>>>>
>>>>> I have also rebooted my entire KVM server and my management server. So far
>>>>> zero luck.
>>>>> On 10/17/22 20:26, Nicolas Vazquez wrote:
>>>>>
>>>>> Can you try taking a manual volume snapshot and share both mgmt and agent logs please?
>>>>>
>>>>> Regards,
>>>>> Nicolas Vazquez
>>>>>
>>>>>
>>>>> From: Granwille Strauss<gr...@namhost.com> <gr...@namhost.com>
>>>>> Date: Monday, 17 October 2022 at 13:15
>>>>> To:users@cloudstack.apache.org <us...@cloudstack.apache.org> <us...@cloudstack.apache.org>
>>>>> Cc: Nicolas Vazquez<Ni...@shapeblue.com> <Ni...@shapeblue.com>
>>>>> Subject: Re: Recurring Snaps Failing
>>>>>
>>>>> Hi
>>>>>
>>>>> I also see another issue with snaps, I deleted all the failed snaps via the cloud management UI. But to my surprise, they're all still at /mnt/primary/snapshots/ see attached screenshot. Something odd is going on with snapshots.
>>>>> On 10/17/22 17:35, Granwille Strauss wrote:
>>>>>
>>>>> Hi
>>>>>
>>>>> I am not 100% if I enabled debug correctly, I followed thishttps://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+agent+debug and ran
>>>>> sed -i 's/INFO/DEBUG/g' /etc/cloudstack/agent/log4j-cloud.xml
>>>>>
>>>>> Attached are the two log files. Same error. I reboot SSVM & vRouter prior no luck.
>>>>> On 10/17/22 16:00, Granwille Strauss wrote:
>>>>>
>>>>> Hi
>>>>>
>>>>> No KVM is on its own server node and management on a different server node. However, management is is primary and secondary storage via NFS. I will enable DEBUG and re-run manual snap to replicate the error.
>>>>> On 10/17/22 15:42, Nicolas Vazquez wrote:
>>>>> Thanks, are the KVM agent and management server running on the same host? Can you increase the log level on the KVM agent logs to DEBUG and retry?
>>>>>
>>>>> Regards,
>>>>> Nicolas Vazquez
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> From: Granwille Strauss<gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com>
>>>>> Date: Monday, 17 October 2022 at 10:04
>>>>> To:users@cloudstack.apache.org<ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org>
>>>>> Cc: Nicolas Vazquez<Ni...@shapeblue.com> <Ni...@shapeblue.com> <Ni...@shapeblue.com>
>>>>> Subject: Re: Recurring Snaps Failing
>>>>>
>>>>> Please find attached TXT
>>>>> On 10/17/22 14:54, Nicolas Vazquez wrote:
>>>>>
>>>>> Ok, seems like the command is not reaching the KVM host, can you share the full management server logs for job-796?
>>>>>
>>>>>
>>>>>
>>>>> Regards,
>>>>>
>>>>> Nicolas Vazquez
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> From: Granwille Strauss<gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com>
>>>>>
>>>>> Date: Monday, 17 October 2022 at 09:32
>>>>>
>>>>> To:users@cloudstack.apache.org<ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org>
>>>>>
>>>>> Cc: Nicolas Vazquez<Ni...@shapeblue.com> <Ni...@shapeblue.com> <Ni...@shapeblue.com>
>>>>>
>>>>> Subject: Re: Recurring Snaps Failing
>>>>>
>>>>>
>>>>>
>>>>> Hi
>>>>>
>>>>>
>>>>>
>>>>> This is all agent threw during the snap process nothing else:
>>>>>
>>>>> 2022-10-17 13:39:12,375 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:a864f9f6) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
>>>>>
>>>>> 2022-10-17 13:40:13,276 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:445216f3) Trying to fetch storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297 from libvirt
>>>>>
>>>>> 2022-10-17 13:40:13,289 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:445216f3) Asking libvirt to refresh storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297
>>>>>
>>>>> 2022-10-17 13:40:13,378 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:445216f3) Trying to fetch storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89 from libvirt
>>>>>
>>>>> 2022-10-17 13:40:13,393 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:445216f3) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
>>>>>
>>>>> 2022-10-17 13:41:13,950 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:87ecfee0) Trying to fetch storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297 from libvirt
>>>>>
>>>>> 2022-10-17 13:41:13,964 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:87ecfee0) Asking libvirt to refresh storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297
>>>>>
>>>>> 2022-10-17 13:41:14,054 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:87ecfee0) Trying to fetch storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89 from libvirt
>>>>>
>>>>> 2022-10-17 13:41:14,067 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:87ecfee0) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
>>>>>
>>>>> 2022-10-17 13:41:57,753 WARN [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-3:null) (logid:d91c35e5) Expected 1 answers while executing SetMonitorServiceCommand but received 3
>>>>>
>>>>> 2022-10-17 13:41:57,938 INFO [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:null) (logid:1549c9a1) Fetching health check result for SEVERIPADDRESS and executing fresh checks: false
>>>>>
>>>>>
>>>>>
>>>>> On 10/17/22 14:29, Nicolas Vazquez wrote:
>>>>>
>>>>>
>>>>>
>>>>> Hi Groete,
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Can you share the CloudStack agent logs from the KVM host?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Regards,
>>>>>
>>>>>
>>>>>
>>>>> Nicolas Vazquez
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> From: Granwille Strauss<gr...@namhost.com.INVALID> <gr...@namhost.com.INVALID> <gr...@namhost.com.INVALID> <gr...@namhost.com.INVALID> <gr...@namhost.com.INVALID>
>>>>>
>>>>>
>>>>>
>>>>> Date: Monday, 17 October 2022 at 08:56
>>>>>
>>>>>
>>>>>
>>>>> To:users@cloudstack.apache.org<ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org>
>>>>>
>>>>>
>>>>>
>>>>> Subject: Re: Recurring Snaps Failing
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Hi
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> I ran a few more tests and can now finally make sense of the failure:
>>>>>
>>>>>
>>>>>
>>>>> 2022-10-17 13:39:08,556 DEBUG [o.a.c.s.d.d.CloudStackPrimaryDataStoreDriverImpl] (Work-Job-Executor-1:ctx-93cb161b job-806/job-807 ctx-3f9ccf45) (logid:8fd10b20) Failed to take snapshot: 23
>>>>>
>>>>>
>>>>>
>>>>> 2022-10-17 13:39:08,558 DEBUG [o.a.c.s.s.SnapshotServiceImpl] (Work-Job-Executor-1:ctx-93cb161b job-806/job-807 ctx-3f9ccf45) (logid:8fd10b20) create snapshot Test001 failed: com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:1, com.cloud.exception.OperationTimedoutException: Commands 7160441932542378205 to Host 1 timed out after 3600
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Seems to be a time out error. But there's no firewalls on the KVM host and NFS server to prevent connections and it worked fine before and randomly just stopped.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> So I still have no idea where to solve this as all other necessary settings are in place. Any ideas?
>>>>>
>>>>>
>>>>>
>>>>> On 10/17/22 08:51, Granwille Strauss wrote:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Hi Guys
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> I enabled recurring snap shots for my Linux VMs and they all seem to be failing with the following error:
>>>>>
>>>>>
>>>>>
>>>>> root@hostname ~ $ grep error /var/log/cloudstack/management/management-server.log
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> 2022-10-17 03:01:29,474 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-27:ctx-99d7d702 job-796 ctx-f26a81fe) (logid:f9a61b00) Failed to create snapshot due to an internal error creating snapshot for volume 635fd5b5-bf14-4cf4-979c-be8a649d5189
>>>>>
>>>>>
>>>>>
>>>>> 2022-10-17 03:01:29,476 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-27:ctx-99d7d702 job-796) (logid:f9a61b00) Complete async job-796, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 635fd5b5-bf14-4cf4-979c-be8a649d5189"}
>>>>>
>>>>>
>>>>>
>>>>> 2022-10-17 03:01:29,478 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-26:ctx-c7865967 job-794 ctx-0700afb0) (logid:5541113c) Failed to create snapshot due to an internal error creating snapshot for volume 70ffef79-3aad-43ab-93d1-153fcd34ad15
>>>>>
>>>>>
>>>>>
>>>>> 2022-10-17 03:01:29,480 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-26:ctx-c7865967 job-794) (logid:5541113c) Complete async job-794, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 70ffef79-3aad-43ab-93d1-153fcd34ad15"}
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> All I am getting out of this is internal error and nothing else. I have checked other logs and I am not quite sure where else to look.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Any assistance in this regard will be appreciated.
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Regards / Groete
>>>>>
>>>>>
>>>>>
>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/>
>>>>>
>>>>>
>>>>>
>>>>> Granwille Strauss // Senior Systems Admin
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> e:granwille@namhost.com<ma...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com>
>>>>>
>>>>>
>>>>>
>>>>> m: +264 81 323 1260<tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260>
>>>>>
>>>>>
>>>>>
>>>>> w:www.namhost.com<http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>>
>>>>>
>>>>>
>>>>> Namhost Internet Services (Pty) Ltd,
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Regards / Groete
>>>>>
>>>>>
>>>>>
>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/>
>>>>>
>>>>>
>>>>>
>>>>> Granwille Strauss // Senior Systems Admin
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> e:granwille@namhost.com<ma...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com>
>>>>>
>>>>>
>>>>>
>>>>> m: +264 81 323 1260<tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260>
>>>>>
>>>>>
>>>>>
>>>>> w:www.namhost.com<http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>>
>>>>>
>>>>>
>>>>> Namhost Internet Services (Pty) Ltd,
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>>
>>>>>
>>>>> Regards / Groete
>>>>>
>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/>
>>>>>
>>>>> Granwille Strauss // Senior Systems Admin
>>>>>
>>>>>
>>>>>
>>>>> e:granwille@namhost.com<ma...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com>
>>>>>
>>>>> m: +264 81 323 1260<tel:+264813231260><tel:+264813231260>
>>>>>
>>>>> w:www.namhost.com<http://www.namhost.com> <http://www.namhost.com><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/>
>>>>>
>>>>>
>>>>>
>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>>
>>>>>
>>>>>
>>>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>>
>>>>> Namhost Internet Services (Pty) Ltd,
>>>>>
>>>>>
>>>>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>>>>
>>>>>
>>>>>
>>>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>>>
>>>>>
>>>>> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Regards / Groete
>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/> <https://www.namhost.com/>
>>>>> Granwille Strauss // Senior Systems Admin
>>>>>
>>>>> e:granwille@namhost.com<ma...@namhost.com> <gr...@namhost.com>
>>>>> m: +264 81 323 1260<tel:+264813231260>
>>>>> w:www.namhost.com<https://www.namhost.com/> <https://www.namhost.com/>
>>>>>
>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost> <https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost> <https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>>
>>>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>> Namhost Internet Services (Pty) Ltd,
>>>>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>>>>
>>>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>>>
>>>>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>>> --
>>>>>
>>>>> Regards / Groete
>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/> <https://www.namhost.com/>
>>>>> Granwille Strauss // Senior Systems Admin
>>>>>
>>>>> e:granwille@namhost.com<ma...@namhost.com> <gr...@namhost.com>
>>>>> m: +264 81 323 1260<tel:+264813231260>
>>>>> w:www.namhost.com<https://www.namhost.com/> <https://www.namhost.com/>
>>>>>
>>>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost> <https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost> <https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>>
>>>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>> Namhost Internet Services (Pty) Ltd,
>>>>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>>>>
>>>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and s
>>>>>
>>>>> --
>>>>> Regards / Groete
>>>>>
>>>>> <https://www.namhost.com> Granwille Strauss // Senior Systems Admin
>>>>>
>>>>> *e:*granwille@namhost.com
>>>>> *m:* +264 81 323 1260 <+264813231260>
>>>>> *w:*www.namhost.com
>>>>>
>>>>> <https://www.facebook.com/namhost> <https://twitter.com/namhost>
>>>>> <https://www.instagram.com/namhostinternetservices/>
>>>>> <https://www.linkedin.com/company/namhos>
>>>>> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>>>
>>>>>
>>>>> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>>>
>>>>> Namhost Internet Services (Pty) Ltd,
>>>>>
>>>>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>>>>
>>>>>
>>>>>
>>>>> The content of this message is confidential. If you have received it by
>>>>> mistake, please inform us by email reply and then delete the message. It is
>>>>> forbidden to copy, forward, or in any way reveal the contents of this
>>>>> message to anyone without our explicit consent. The integrity and security
>>>>> of this email cannot be guaranteed over the Internet. Therefore, the sender
>>>>> will not be held liable for any damage caused by the message. For our full
>>>>> privacy policy and disclaimers, please go tohttps://www.namhost.com/
>>>>> privacy-policy
>>>>>
>>>>> [image: Powered by AdSigner]
>>>>> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>>>
>>> --
>>> Regards / Groete
>>>
>>> <https://www.namhost.com> Granwille Strauss // Senior Systems Admin
>>>
>>> *e:* granwille@namhost.com
>>> *m:* +264 81 323 1260 <tel:+264813231260>
>>> *w:* www.namhost.com <https://www.namhost.com/>
>>>
>>> <https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>
>>> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>
>>> Namhost Internet Services (Pty) Ltd,
>>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>>
>>>
>>> The content of this message is confidential. If you have received it
>>> by mistake, please inform us by email reply and then delete the
>>> message. It is forbidden to copy, forward, or in any way reveal the
>>> contents of this message to anyone without our explicit consent. The
>>> integrity and security of this email cannot be guaranteed over the
>>> Internet. Therefore, the sender will not be held liable for any
>>> damage caused by the message. For our full privacy policy and
>>> disclaimers, please go to https://www.namhost.com/privacy-policy
>>>
>>> Powered by AdSigner
>>> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>> --
>> Regards / Groete
>>
>> <https://www.namhost.com> Granwille Strauss // Senior Systems Admin
>>
>> *e:* granwille@namhost.com
>> *m:* +264 81 323 1260 <tel:+264813231260>
>> *w:* www.namhost.com <https://www.namhost.com/>
>>
>> <https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>
>> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>
>> Namhost Internet Services (Pty) Ltd,
>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>
>>
>> The content of this message is confidential. If you have received it
>> by mistake, please inform us by email reply and then delete the
>> message. It is forbidden to copy, forward, or in any way reveal the
>> contents of this message to anyone without our explicit consent. The
>> integrity and security of this email cannot be guaranteed over the
>> Internet. Therefore, the sender will not be held liable for any
>> damage caused by the message. For our full privacy policy and
>> disclaimers, please go to https://www.namhost.com/privacy-policy
>>
>> Powered by AdSigner
>> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
> --
> Regards / Groete
>
> <https://www.namhost.com> Granwille Strauss // Senior Systems Admin
>
> *e:* granwille@namhost.com
> *m:* +264 81 323 1260 <tel:+264813231260>
> *w:* www.namhost.com <https://www.namhost.com/>
>
> <https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>
> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>
> Namhost Internet Services (Pty) Ltd,
> 24 Black Eagle Rd, Hermanus, 7210, RSA
>
>
> The content of this message is confidential. If you have received it
> by mistake, please inform us by email reply and then delete the
> message. It is forbidden to copy, forward, or in any way reveal the
> contents of this message to anyone without our explicit consent. The
> integrity and security of this email cannot be guaranteed over the
> Internet. Therefore, the sender will not be held liable for any damage
> caused by the message. For our full privacy policy and disclaimers,
> please go to https://www.namhost.com/privacy-policy
>
> Powered by AdSigner
> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
--
Regards / Groete
<https://www.namhost.com> Granwille Strauss // Senior Systems Admin
*e:* granwille@namhost.com
*m:* +264 81 323 1260 <tel:+264813231260>
*w:* www.namhost.com <https://www.namhost.com/>
<https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
Namhost Internet Services (Pty) Ltd,
24 Black Eagle Rd, Hermanus, 7210, RSA
The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It
is forbidden to copy, forward, or in any way reveal the contents of this
message to anyone without our explicit consent. The integrity and
security of this email cannot be guaranteed over the Internet.
Therefore, the sender will not be held liable for any damage caused by
the message. For our full privacy policy and disclaimers, please go to
https://www.namhost.com/privacy-policy
Powered by AdSigner
<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
Re: Recurring Snaps Failing
Posted by Granwille Strauss <gr...@namhost.com.INVALID>.
Here's the last 10 minutes before the error is triggered in the logs,
this was the test run before switching off VM. That test is still
running and will provide its logs shortly, however, I am confident, it
will show same results.
On 10/17/22 21:22, Granwille Strauss wrote:
>
> Hi Wei
>
> Please see attached. Please keep in mind, this worked 100% a few days
> ago when I tested on VM recurring snaps and it was perfect then. So I
> enabled recurring snaps for all 10 linux VMs I host. And I set them
> all to daily at 2AM.
> They failed after the first day. So I figured maybe too many
> concurrent jobs running at the same time.
>
> So I split them all to once weekly at different days at different
> hours. And still it failed. And now when doing it manually it fails
> too. But for some reason the snap ends up in /primary location is just
> returns an error for some reason.
>
> On 10/17/22 21:13, Wei ZHOU wrote:
>> Hi,
>>
>> What is the kvm host os and version ?
>>
>> -Wei
>>
>> On Monday, 17 October 2022, Granwille Strauss<gr...@namhost.com.invalid>
>> wrote:
>>
>>> The error seems to happen exactly at 1 hour intervals. All of them have
>>> failed exactly at one hour. Also, these are all volume snapshots that I am
>>> taking, so the logs I sent you so far are all in this regard. They are not
>>> VM snaphots, nor are they powered off. I am running a new volume snap shot
>>> again, this time VM powered off lets see.
>>>
>>> I have also rebooted my entire KVM server and my management server. So far
>>> zero luck.
>>> On 10/17/22 20:26, Nicolas Vazquez wrote:
>>>
>>> Can you try taking a manual volume snapshot and share both mgmt and agent logs please?
>>>
>>> Regards,
>>> Nicolas Vazquez
>>>
>>>
>>> From: Granwille Strauss<gr...@namhost.com> <gr...@namhost.com>
>>> Date: Monday, 17 October 2022 at 13:15
>>> To:users@cloudstack.apache.org <us...@cloudstack.apache.org> <us...@cloudstack.apache.org>
>>> Cc: Nicolas Vazquez<Ni...@shapeblue.com> <Ni...@shapeblue.com>
>>> Subject: Re: Recurring Snaps Failing
>>>
>>> Hi
>>>
>>> I also see another issue with snaps, I deleted all the failed snaps via the cloud management UI. But to my surprise, they're all still at /mnt/primary/snapshots/ see attached screenshot. Something odd is going on with snapshots.
>>> On 10/17/22 17:35, Granwille Strauss wrote:
>>>
>>> Hi
>>>
>>> I am not 100% if I enabled debug correctly, I followed thishttps://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+agent+debug and ran
>>> sed -i 's/INFO/DEBUG/g' /etc/cloudstack/agent/log4j-cloud.xml
>>>
>>> Attached are the two log files. Same error. I reboot SSVM & vRouter prior no luck.
>>> On 10/17/22 16:00, Granwille Strauss wrote:
>>>
>>> Hi
>>>
>>> No KVM is on its own server node and management on a different server node. However, management is is primary and secondary storage via NFS. I will enable DEBUG and re-run manual snap to replicate the error.
>>> On 10/17/22 15:42, Nicolas Vazquez wrote:
>>> Thanks, are the KVM agent and management server running on the same host? Can you increase the log level on the KVM agent logs to DEBUG and retry?
>>>
>>> Regards,
>>> Nicolas Vazquez
>>>
>>>
>>>
>>>
>>>
>>>
>>> From: Granwille Strauss<gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com>
>>> Date: Monday, 17 October 2022 at 10:04
>>> To:users@cloudstack.apache.org<ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org>
>>> Cc: Nicolas Vazquez<Ni...@shapeblue.com> <Ni...@shapeblue.com> <Ni...@shapeblue.com>
>>> Subject: Re: Recurring Snaps Failing
>>>
>>> Please find attached TXT
>>> On 10/17/22 14:54, Nicolas Vazquez wrote:
>>>
>>> Ok, seems like the command is not reaching the KVM host, can you share the full management server logs for job-796?
>>>
>>>
>>>
>>> Regards,
>>>
>>> Nicolas Vazquez
>>>
>>>
>>>
>>>
>>>
>>> From: Granwille Strauss<gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com>
>>>
>>> Date: Monday, 17 October 2022 at 09:32
>>>
>>> To:users@cloudstack.apache.org<ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org>
>>>
>>> Cc: Nicolas Vazquez<Ni...@shapeblue.com> <Ni...@shapeblue.com> <Ni...@shapeblue.com>
>>>
>>> Subject: Re: Recurring Snaps Failing
>>>
>>>
>>>
>>> Hi
>>>
>>>
>>>
>>> This is all agent threw during the snap process nothing else:
>>>
>>> 2022-10-17 13:39:12,375 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:a864f9f6) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
>>>
>>> 2022-10-17 13:40:13,276 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:445216f3) Trying to fetch storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297 from libvirt
>>>
>>> 2022-10-17 13:40:13,289 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:445216f3) Asking libvirt to refresh storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297
>>>
>>> 2022-10-17 13:40:13,378 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:445216f3) Trying to fetch storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89 from libvirt
>>>
>>> 2022-10-17 13:40:13,393 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:445216f3) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
>>>
>>> 2022-10-17 13:41:13,950 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:87ecfee0) Trying to fetch storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297 from libvirt
>>>
>>> 2022-10-17 13:41:13,964 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:87ecfee0) Asking libvirt to refresh storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297
>>>
>>> 2022-10-17 13:41:14,054 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:87ecfee0) Trying to fetch storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89 from libvirt
>>>
>>> 2022-10-17 13:41:14,067 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:87ecfee0) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
>>>
>>> 2022-10-17 13:41:57,753 WARN [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-3:null) (logid:d91c35e5) Expected 1 answers while executing SetMonitorServiceCommand but received 3
>>>
>>> 2022-10-17 13:41:57,938 INFO [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:null) (logid:1549c9a1) Fetching health check result for SEVERIPADDRESS and executing fresh checks: false
>>>
>>>
>>>
>>> On 10/17/22 14:29, Nicolas Vazquez wrote:
>>>
>>>
>>>
>>> Hi Groete,
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Can you share the CloudStack agent logs from the KVM host?
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Regards,
>>>
>>>
>>>
>>> Nicolas Vazquez
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> From: Granwille Strauss<gr...@namhost.com.INVALID> <gr...@namhost.com.INVALID> <gr...@namhost.com.INVALID> <gr...@namhost.com.INVALID> <gr...@namhost.com.INVALID>
>>>
>>>
>>>
>>> Date: Monday, 17 October 2022 at 08:56
>>>
>>>
>>>
>>> To:users@cloudstack.apache.org<ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org>
>>>
>>>
>>>
>>> Subject: Re: Recurring Snaps Failing
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Hi
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> I ran a few more tests and can now finally make sense of the failure:
>>>
>>>
>>>
>>> 2022-10-17 13:39:08,556 DEBUG [o.a.c.s.d.d.CloudStackPrimaryDataStoreDriverImpl] (Work-Job-Executor-1:ctx-93cb161b job-806/job-807 ctx-3f9ccf45) (logid:8fd10b20) Failed to take snapshot: 23
>>>
>>>
>>>
>>> 2022-10-17 13:39:08,558 DEBUG [o.a.c.s.s.SnapshotServiceImpl] (Work-Job-Executor-1:ctx-93cb161b job-806/job-807 ctx-3f9ccf45) (logid:8fd10b20) create snapshot Test001 failed: com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:1, com.cloud.exception.OperationTimedoutException: Commands 7160441932542378205 to Host 1 timed out after 3600
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Seems to be a time out error. But there's no firewalls on the KVM host and NFS server to prevent connections and it worked fine before and randomly just stopped.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> So I still have no idea where to solve this as all other necessary settings are in place. Any ideas?
>>>
>>>
>>>
>>> On 10/17/22 08:51, Granwille Strauss wrote:
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Hi Guys
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> I enabled recurring snap shots for my Linux VMs and they all seem to be failing with the following error:
>>>
>>>
>>>
>>> root@hostname ~ $ grep error /var/log/cloudstack/management/management-server.log
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> 2022-10-17 03:01:29,474 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-27:ctx-99d7d702 job-796 ctx-f26a81fe) (logid:f9a61b00) Failed to create snapshot due to an internal error creating snapshot for volume 635fd5b5-bf14-4cf4-979c-be8a649d5189
>>>
>>>
>>>
>>> 2022-10-17 03:01:29,476 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-27:ctx-99d7d702 job-796) (logid:f9a61b00) Complete async job-796, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 635fd5b5-bf14-4cf4-979c-be8a649d5189"}
>>>
>>>
>>>
>>> 2022-10-17 03:01:29,478 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-26:ctx-c7865967 job-794 ctx-0700afb0) (logid:5541113c) Failed to create snapshot due to an internal error creating snapshot for volume 70ffef79-3aad-43ab-93d1-153fcd34ad15
>>>
>>>
>>>
>>> 2022-10-17 03:01:29,480 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-26:ctx-c7865967 job-794) (logid:5541113c) Complete async job-794, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 70ffef79-3aad-43ab-93d1-153fcd34ad15"}
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> All I am getting out of this is internal error and nothing else. I have checked other logs and I am not quite sure where else to look.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Any assistance in this regard will be appreciated.
>>>
>>>
>>>
>>> --
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Regards / Groete
>>>
>>>
>>>
>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/>
>>>
>>>
>>>
>>> Granwille Strauss // Senior Systems Admin
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> e:granwille@namhost.com<ma...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com>
>>>
>>>
>>>
>>> m: +264 81 323 1260<tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260>
>>>
>>>
>>>
>>> w:www.namhost.com<http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>
>>>
>>>
>>> Namhost Internet Services (Pty) Ltd,
>>>
>>>
>>>
>>>
>>>
>>>
>>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>
>>>
>>>
>>> --
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Regards / Groete
>>>
>>>
>>>
>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/>
>>>
>>>
>>>
>>> Granwille Strauss // Senior Systems Admin
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> e:granwille@namhost.com<ma...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com>
>>>
>>>
>>>
>>> m: +264 81 323 1260<tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260>
>>>
>>>
>>>
>>> w:www.namhost.com<http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>
>>>
>>>
>>> Namhost Internet Services (Pty) Ltd,
>>>
>>>
>>>
>>>
>>>
>>>
>>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>>
>>>
>>>
>>> Regards / Groete
>>>
>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/>
>>>
>>> Granwille Strauss // Senior Systems Admin
>>>
>>>
>>>
>>> e:granwille@namhost.com<ma...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com>
>>>
>>> m: +264 81 323 1260<tel:+264813231260><tel:+264813231260>
>>>
>>> w:www.namhost.com<http://www.namhost.com> <http://www.namhost.com><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/>
>>>
>>>
>>>
>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>
>>>
>>>
>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>
>>> Namhost Internet Services (Pty) Ltd,
>>>
>>>
>>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>>
>>>
>>>
>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>
>>>
>>> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>> Regards / Groete
>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/> <https://www.namhost.com/>
>>> Granwille Strauss // Senior Systems Admin
>>>
>>> e:granwille@namhost.com<ma...@namhost.com> <gr...@namhost.com>
>>> m: +264 81 323 1260<tel:+264813231260>
>>> w:www.namhost.com<https://www.namhost.com/> <https://www.namhost.com/>
>>>
>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost> <https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost> <https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>
>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>> Namhost Internet Services (Pty) Ltd,
>>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>>
>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>
>>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>> --
>>>
>>> Regards / Groete
>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/> <https://www.namhost.com/>
>>> Granwille Strauss // Senior Systems Admin
>>>
>>> e:granwille@namhost.com<ma...@namhost.com> <gr...@namhost.com>
>>> m: +264 81 323 1260<tel:+264813231260>
>>> w:www.namhost.com<https://www.namhost.com/> <https://www.namhost.com/>
>>>
>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost> <https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost> <https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>
>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>> Namhost Internet Services (Pty) Ltd,
>>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>>
>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and s
>>>
>>> --
>>> Regards / Groete
>>>
>>> <https://www.namhost.com> Granwille Strauss // Senior Systems Admin
>>>
>>> *e:*granwille@namhost.com
>>> *m:* +264 81 323 1260 <+264813231260>
>>> *w:*www.namhost.com
>>>
>>> <https://www.facebook.com/namhost> <https://twitter.com/namhost>
>>> <https://www.instagram.com/namhostinternetservices/>
>>> <https://www.linkedin.com/company/namhos>
>>> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>
>>>
>>> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>
>>> Namhost Internet Services (Pty) Ltd,
>>>
>>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>>
>>>
>>>
>>> The content of this message is confidential. If you have received it by
>>> mistake, please inform us by email reply and then delete the message. It is
>>> forbidden to copy, forward, or in any way reveal the contents of this
>>> message to anyone without our explicit consent. The integrity and security
>>> of this email cannot be guaranteed over the Internet. Therefore, the sender
>>> will not be held liable for any damage caused by the message. For our full
>>> privacy policy and disclaimers, please go tohttps://www.namhost.com/
>>> privacy-policy
>>>
>>> [image: Powered by AdSigner]
>>> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>
> --
> Regards / Groete
>
> <https://www.namhost.com> Granwille Strauss // Senior Systems Admin
>
> *e:* granwille@namhost.com
> *m:* +264 81 323 1260 <tel:+264813231260>
> *w:* www.namhost.com <https://www.namhost.com/>
>
> <https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>
> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>
> Namhost Internet Services (Pty) Ltd,
> 24 Black Eagle Rd, Hermanus, 7210, RSA
>
>
> The content of this message is confidential. If you have received it
> by mistake, please inform us by email reply and then delete the
> message. It is forbidden to copy, forward, or in any way reveal the
> contents of this message to anyone without our explicit consent. The
> integrity and security of this email cannot be guaranteed over the
> Internet. Therefore, the sender will not be held liable for any damage
> caused by the message. For our full privacy policy and disclaimers,
> please go to https://www.namhost.com/privacy-policy
>
> Powered by AdSigner
> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
--
Regards / Groete
<https://www.namhost.com> Granwille Strauss // Senior Systems Admin
*e:* granwille@namhost.com
*m:* +264 81 323 1260 <tel:+264813231260>
*w:* www.namhost.com <https://www.namhost.com/>
<https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
Namhost Internet Services (Pty) Ltd,
24 Black Eagle Rd, Hermanus, 7210, RSA
The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It
is forbidden to copy, forward, or in any way reveal the contents of this
message to anyone without our explicit consent. The integrity and
security of this email cannot be guaranteed over the Internet.
Therefore, the sender will not be held liable for any damage caused by
the message. For our full privacy policy and disclaimers, please go to
https://www.namhost.com/privacy-policy
Powered by AdSigner
<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
Re: Recurring Snaps Failing
Posted by Granwille Strauss <gr...@namhost.com.INVALID>.
Hi Wei
Please see attached. Please keep in mind, this worked 100% a few days
ago when I tested on VM recurring snaps and it was perfect then. So I
enabled recurring snaps for all 10 linux VMs I host. And I set them all
to daily at 2AM.
They failed after the first day. So I figured maybe too many concurrent
jobs running at the same time.
So I split them all to once weekly at different days at different
hours. And still it failed. And now when doing it manually it fails too.
But for some reason the snap ends up in /primary location is just
returns an error for some reason.
On 10/17/22 21:13, Wei ZHOU wrote:
> Hi,
>
> What is the kvm host os and version ?
>
> -Wei
>
> On Monday, 17 October 2022, Granwille Strauss<gr...@namhost.com.invalid>
> wrote:
>
>> The error seems to happen exactly at 1 hour intervals. All of them have
>> failed exactly at one hour. Also, these are all volume snapshots that I am
>> taking, so the logs I sent you so far are all in this regard. They are not
>> VM snaphots, nor are they powered off. I am running a new volume snap shot
>> again, this time VM powered off lets see.
>>
>> I have also rebooted my entire KVM server and my management server. So far
>> zero luck.
>> On 10/17/22 20:26, Nicolas Vazquez wrote:
>>
>> Can you try taking a manual volume snapshot and share both mgmt and agent logs please?
>>
>> Regards,
>> Nicolas Vazquez
>>
>>
>> From: Granwille Strauss<gr...@namhost.com> <gr...@namhost.com>
>> Date: Monday, 17 October 2022 at 13:15
>> To:users@cloudstack.apache.org <us...@cloudstack.apache.org> <us...@cloudstack.apache.org>
>> Cc: Nicolas Vazquez<Ni...@shapeblue.com> <Ni...@shapeblue.com>
>> Subject: Re: Recurring Snaps Failing
>>
>> Hi
>>
>> I also see another issue with snaps, I deleted all the failed snaps via the cloud management UI. But to my surprise, they're all still at /mnt/primary/snapshots/ see attached screenshot. Something odd is going on with snapshots.
>> On 10/17/22 17:35, Granwille Strauss wrote:
>>
>> Hi
>>
>> I am not 100% if I enabled debug correctly, I followed thishttps://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+agent+debug and ran
>> sed -i 's/INFO/DEBUG/g' /etc/cloudstack/agent/log4j-cloud.xml
>>
>> Attached are the two log files. Same error. I reboot SSVM & vRouter prior no luck.
>> On 10/17/22 16:00, Granwille Strauss wrote:
>>
>> Hi
>>
>> No KVM is on its own server node and management on a different server node. However, management is is primary and secondary storage via NFS. I will enable DEBUG and re-run manual snap to replicate the error.
>> On 10/17/22 15:42, Nicolas Vazquez wrote:
>> Thanks, are the KVM agent and management server running on the same host? Can you increase the log level on the KVM agent logs to DEBUG and retry?
>>
>> Regards,
>> Nicolas Vazquez
>>
>>
>>
>>
>>
>>
>> From: Granwille Strauss<gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com>
>> Date: Monday, 17 October 2022 at 10:04
>> To:users@cloudstack.apache.org<ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org>
>> Cc: Nicolas Vazquez<Ni...@shapeblue.com> <Ni...@shapeblue.com> <Ni...@shapeblue.com>
>> Subject: Re: Recurring Snaps Failing
>>
>> Please find attached TXT
>> On 10/17/22 14:54, Nicolas Vazquez wrote:
>>
>> Ok, seems like the command is not reaching the KVM host, can you share the full management server logs for job-796?
>>
>>
>>
>> Regards,
>>
>> Nicolas Vazquez
>>
>>
>>
>>
>>
>> From: Granwille Strauss<gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com>
>>
>> Date: Monday, 17 October 2022 at 09:32
>>
>> To:users@cloudstack.apache.org<ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org>
>>
>> Cc: Nicolas Vazquez<Ni...@shapeblue.com> <Ni...@shapeblue.com> <Ni...@shapeblue.com>
>>
>> Subject: Re: Recurring Snaps Failing
>>
>>
>>
>> Hi
>>
>>
>>
>> This is all agent threw during the snap process nothing else:
>>
>> 2022-10-17 13:39:12,375 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:a864f9f6) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
>>
>> 2022-10-17 13:40:13,276 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:445216f3) Trying to fetch storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297 from libvirt
>>
>> 2022-10-17 13:40:13,289 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:445216f3) Asking libvirt to refresh storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297
>>
>> 2022-10-17 13:40:13,378 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:445216f3) Trying to fetch storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89 from libvirt
>>
>> 2022-10-17 13:40:13,393 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:445216f3) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
>>
>> 2022-10-17 13:41:13,950 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:87ecfee0) Trying to fetch storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297 from libvirt
>>
>> 2022-10-17 13:41:13,964 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:87ecfee0) Asking libvirt to refresh storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297
>>
>> 2022-10-17 13:41:14,054 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:87ecfee0) Trying to fetch storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89 from libvirt
>>
>> 2022-10-17 13:41:14,067 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:87ecfee0) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
>>
>> 2022-10-17 13:41:57,753 WARN [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-3:null) (logid:d91c35e5) Expected 1 answers while executing SetMonitorServiceCommand but received 3
>>
>> 2022-10-17 13:41:57,938 INFO [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:null) (logid:1549c9a1) Fetching health check result for SEVERIPADDRESS and executing fresh checks: false
>>
>>
>>
>> On 10/17/22 14:29, Nicolas Vazquez wrote:
>>
>>
>>
>> Hi Groete,
>>
>>
>>
>>
>>
>>
>>
>> Can you share the CloudStack agent logs from the KVM host?
>>
>>
>>
>>
>>
>>
>>
>> Regards,
>>
>>
>>
>> Nicolas Vazquez
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> From: Granwille Strauss<gr...@namhost.com.INVALID> <gr...@namhost.com.INVALID> <gr...@namhost.com.INVALID> <gr...@namhost.com.INVALID> <gr...@namhost.com.INVALID>
>>
>>
>>
>> Date: Monday, 17 October 2022 at 08:56
>>
>>
>>
>> To:users@cloudstack.apache.org<ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org>
>>
>>
>>
>> Subject: Re: Recurring Snaps Failing
>>
>>
>>
>>
>>
>>
>>
>> Hi
>>
>>
>>
>>
>>
>>
>>
>> I ran a few more tests and can now finally make sense of the failure:
>>
>>
>>
>> 2022-10-17 13:39:08,556 DEBUG [o.a.c.s.d.d.CloudStackPrimaryDataStoreDriverImpl] (Work-Job-Executor-1:ctx-93cb161b job-806/job-807 ctx-3f9ccf45) (logid:8fd10b20) Failed to take snapshot: 23
>>
>>
>>
>> 2022-10-17 13:39:08,558 DEBUG [o.a.c.s.s.SnapshotServiceImpl] (Work-Job-Executor-1:ctx-93cb161b job-806/job-807 ctx-3f9ccf45) (logid:8fd10b20) create snapshot Test001 failed: com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:1, com.cloud.exception.OperationTimedoutException: Commands 7160441932542378205 to Host 1 timed out after 3600
>>
>>
>>
>>
>>
>>
>>
>> Seems to be a time out error. But there's no firewalls on the KVM host and NFS server to prevent connections and it worked fine before and randomly just stopped.
>>
>>
>>
>>
>>
>>
>>
>> So I still have no idea where to solve this as all other necessary settings are in place. Any ideas?
>>
>>
>>
>> On 10/17/22 08:51, Granwille Strauss wrote:
>>
>>
>>
>>
>>
>>
>>
>> Hi Guys
>>
>>
>>
>>
>>
>>
>>
>> I enabled recurring snap shots for my Linux VMs and they all seem to be failing with the following error:
>>
>>
>>
>> root@hostname ~ $ grep error /var/log/cloudstack/management/management-server.log
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> 2022-10-17 03:01:29,474 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-27:ctx-99d7d702 job-796 ctx-f26a81fe) (logid:f9a61b00) Failed to create snapshot due to an internal error creating snapshot for volume 635fd5b5-bf14-4cf4-979c-be8a649d5189
>>
>>
>>
>> 2022-10-17 03:01:29,476 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-27:ctx-99d7d702 job-796) (logid:f9a61b00) Complete async job-796, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 635fd5b5-bf14-4cf4-979c-be8a649d5189"}
>>
>>
>>
>> 2022-10-17 03:01:29,478 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-26:ctx-c7865967 job-794 ctx-0700afb0) (logid:5541113c) Failed to create snapshot due to an internal error creating snapshot for volume 70ffef79-3aad-43ab-93d1-153fcd34ad15
>>
>>
>>
>> 2022-10-17 03:01:29,480 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-26:ctx-c7865967 job-794) (logid:5541113c) Complete async job-794, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 70ffef79-3aad-43ab-93d1-153fcd34ad15"}
>>
>>
>>
>>
>>
>>
>>
>> All I am getting out of this is internal error and nothing else. I have checked other logs and I am not quite sure where else to look.
>>
>>
>>
>>
>>
>>
>>
>> Any assistance in this regard will be appreciated.
>>
>>
>>
>> --
>>
>>
>>
>>
>>
>>
>>
>> Regards / Groete
>>
>>
>>
>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/>
>>
>>
>>
>> Granwille Strauss // Senior Systems Admin
>>
>>
>>
>>
>>
>>
>>
>> e:granwille@namhost.com<ma...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com>
>>
>>
>>
>> m: +264 81 323 1260<tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260>
>>
>>
>>
>> w:www.namhost.com<http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/>
>>
>>
>>
>>
>>
>>
>>
>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>
>>
>>
>>
>>
>>
>>
>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>
>>
>>
>> Namhost Internet Services (Pty) Ltd,
>>
>>
>>
>>
>>
>>
>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>
>>
>>
>>
>>
>>
>>
>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>
>>
>>
>>
>>
>>
>>
>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>
>>
>>
>> --
>>
>>
>>
>>
>>
>>
>>
>> Regards / Groete
>>
>>
>>
>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/>
>>
>>
>>
>> Granwille Strauss // Senior Systems Admin
>>
>>
>>
>>
>>
>>
>>
>> e:granwille@namhost.com<ma...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com>
>>
>>
>>
>> m: +264 81 323 1260<tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260>
>>
>>
>>
>> w:www.namhost.com<http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/>
>>
>>
>>
>>
>>
>>
>>
>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>
>>
>>
>>
>>
>>
>>
>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>
>>
>>
>> Namhost Internet Services (Pty) Ltd,
>>
>>
>>
>>
>>
>>
>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>
>>
>>
>>
>>
>>
>>
>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>
>>
>>
>>
>>
>>
>>
>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> --
>>
>>
>>
>> Regards / Groete
>>
>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/>
>>
>> Granwille Strauss // Senior Systems Admin
>>
>>
>>
>> e:granwille@namhost.com<ma...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com>
>>
>> m: +264 81 323 1260<tel:+264813231260><tel:+264813231260>
>>
>> w:www.namhost.com<http://www.namhost.com> <http://www.namhost.com><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/>
>>
>>
>>
>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>
>>
>>
>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>
>> Namhost Internet Services (Pty) Ltd,
>>
>>
>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>
>>
>>
>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>
>>
>> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>
>>
>>
>>
>>
>>
>>
>>
>> --
>> Regards / Groete
>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/> <https://www.namhost.com/>
>> Granwille Strauss // Senior Systems Admin
>>
>> e:granwille@namhost.com<ma...@namhost.com> <gr...@namhost.com>
>> m: +264 81 323 1260<tel:+264813231260>
>> w:www.namhost.com<https://www.namhost.com/> <https://www.namhost.com/>
>>
>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost> <https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost> <https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>
>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>> Namhost Internet Services (Pty) Ltd,
>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>
>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>
>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>> --
>>
>> Regards / Groete
>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/> <https://www.namhost.com/>
>> Granwille Strauss // Senior Systems Admin
>>
>> e:granwille@namhost.com<ma...@namhost.com> <gr...@namhost.com>
>> m: +264 81 323 1260<tel:+264813231260>
>> w:www.namhost.com<https://www.namhost.com/> <https://www.namhost.com/>
>>
>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost> <https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost> <https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>
>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>> Namhost Internet Services (Pty) Ltd,
>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>
>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and s
>>
>> --
>> Regards / Groete
>>
>> <https://www.namhost.com> Granwille Strauss // Senior Systems Admin
>>
>> *e:*granwille@namhost.com
>> *m:* +264 81 323 1260 <+264813231260>
>> *w:*www.namhost.com
>>
>> <https://www.facebook.com/namhost> <https://twitter.com/namhost>
>> <https://www.instagram.com/namhostinternetservices/>
>> <https://www.linkedin.com/company/namhos>
>> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>
>>
>> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>
>> Namhost Internet Services (Pty) Ltd,
>>
>> 24 Black Eagle Rd, Hermanus<https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>>
>>
>>
>> The content of this message is confidential. If you have received it by
>> mistake, please inform us by email reply and then delete the message. It is
>> forbidden to copy, forward, or in any way reveal the contents of this
>> message to anyone without our explicit consent. The integrity and security
>> of this email cannot be guaranteed over the Internet. Therefore, the sender
>> will not be held liable for any damage caused by the message. For our full
>> privacy policy and disclaimers, please go tohttps://www.namhost.com/
>> privacy-policy
>>
>> [image: Powered by AdSigner]
>> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>
--
Regards / Groete
<https://www.namhost.com> Granwille Strauss // Senior Systems Admin
*e:* granwille@namhost.com
*m:* +264 81 323 1260 <tel:+264813231260>
*w:* www.namhost.com <https://www.namhost.com/>
<https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
Namhost Internet Services (Pty) Ltd,
24 Black Eagle Rd, Hermanus, 7210, RSA
The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It
is forbidden to copy, forward, or in any way reveal the contents of this
message to anyone without our explicit consent. The integrity and
security of this email cannot be guaranteed over the Internet.
Therefore, the sender will not be held liable for any damage caused by
the message. For our full privacy policy and disclaimers, please go to
https://www.namhost.com/privacy-policy
Powered by AdSigner
<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
Re: Recurring Snaps Failing
Posted by Wei ZHOU <us...@gmail.com>.
Hi,
What is the kvm host os and version ?
-Wei
On Monday, 17 October 2022, Granwille Strauss <gr...@namhost.com.invalid>
wrote:
> The error seems to happen exactly at 1 hour intervals. All of them have
> failed exactly at one hour. Also, these are all volume snapshots that I am
> taking, so the logs I sent you so far are all in this regard. They are not
> VM snaphots, nor are they powered off. I am running a new volume snap shot
> again, this time VM powered off lets see.
>
> I have also rebooted my entire KVM server and my management server. So far
> zero luck.
> On 10/17/22 20:26, Nicolas Vazquez wrote:
>
> Can you try taking a manual volume snapshot and share both mgmt and agent logs please?
>
> Regards,
> Nicolas Vazquez
>
>
> From: Granwille Strauss <gr...@namhost.com> <gr...@namhost.com>
> Date: Monday, 17 October 2022 at 13:15
> To: users@cloudstack.apache.org <us...@cloudstack.apache.org> <us...@cloudstack.apache.org>
> Cc: Nicolas Vazquez <Ni...@shapeblue.com> <Ni...@shapeblue.com>
> Subject: Re: Recurring Snaps Failing
>
> Hi
>
> I also see another issue with snaps, I deleted all the failed snaps via the cloud management UI. But to my surprise, they're all still at /mnt/primary/snapshots/ see attached screenshot. Something odd is going on with snapshots.
> On 10/17/22 17:35, Granwille Strauss wrote:
>
> Hi
>
> I am not 100% if I enabled debug correctly, I followed this https://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+agent+debug and ran
> sed -i 's/INFO/DEBUG/g' /etc/cloudstack/agent/log4j-cloud.xml
>
> Attached are the two log files. Same error. I reboot SSVM & vRouter prior no luck.
> On 10/17/22 16:00, Granwille Strauss wrote:
>
> Hi
>
> No KVM is on its own server node and management on a different server node. However, management is is primary and secondary storage via NFS. I will enable DEBUG and re-run manual snap to replicate the error.
> On 10/17/22 15:42, Nicolas Vazquez wrote:
> Thanks, are the KVM agent and management server running on the same host? Can you increase the log level on the KVM agent logs to DEBUG and retry?
>
> Regards,
> Nicolas Vazquez
>
>
>
>
>
>
> From: Granwille Strauss <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com>
> Date: Monday, 17 October 2022 at 10:04
> To: users@cloudstack.apache.org<ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org>
> Cc: Nicolas Vazquez <Ni...@shapeblue.com> <Ni...@shapeblue.com> <Ni...@shapeblue.com>
> Subject: Re: Recurring Snaps Failing
>
> Please find attached TXT
> On 10/17/22 14:54, Nicolas Vazquez wrote:
>
> Ok, seems like the command is not reaching the KVM host, can you share the full management server logs for job-796?
>
>
>
> Regards,
>
> Nicolas Vazquez
>
>
>
>
>
> From: Granwille Strauss <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com>
>
> Date: Monday, 17 October 2022 at 09:32
>
> To: users@cloudstack.apache.org<ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org>
>
> Cc: Nicolas Vazquez <Ni...@shapeblue.com> <Ni...@shapeblue.com> <Ni...@shapeblue.com>
>
> Subject: Re: Recurring Snaps Failing
>
>
>
> Hi
>
>
>
> This is all agent threw during the snap process nothing else:
>
> 2022-10-17 13:39:12,375 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:a864f9f6) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
>
> 2022-10-17 13:40:13,276 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:445216f3) Trying to fetch storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297 from libvirt
>
> 2022-10-17 13:40:13,289 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:445216f3) Asking libvirt to refresh storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297
>
> 2022-10-17 13:40:13,378 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:445216f3) Trying to fetch storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89 from libvirt
>
> 2022-10-17 13:40:13,393 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:445216f3) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
>
> 2022-10-17 13:41:13,950 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:87ecfee0) Trying to fetch storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297 from libvirt
>
> 2022-10-17 13:41:13,964 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:87ecfee0) Asking libvirt to refresh storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297
>
> 2022-10-17 13:41:14,054 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:87ecfee0) Trying to fetch storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89 from libvirt
>
> 2022-10-17 13:41:14,067 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:87ecfee0) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
>
> 2022-10-17 13:41:57,753 WARN [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-3:null) (logid:d91c35e5) Expected 1 answers while executing SetMonitorServiceCommand but received 3
>
> 2022-10-17 13:41:57,938 INFO [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:null) (logid:1549c9a1) Fetching health check result for SEVERIPADDRESS and executing fresh checks: false
>
>
>
> On 10/17/22 14:29, Nicolas Vazquez wrote:
>
>
>
> Hi Groete,
>
>
>
>
>
>
>
> Can you share the CloudStack agent logs from the KVM host?
>
>
>
>
>
>
>
> Regards,
>
>
>
> Nicolas Vazquez
>
>
>
>
>
>
>
>
>
>
>
> From: Granwille Strauss <gr...@namhost.com.INVALID> <gr...@namhost.com.INVALID> <gr...@namhost.com.INVALID> <gr...@namhost.com.INVALID> <gr...@namhost.com.INVALID>
>
>
>
> Date: Monday, 17 October 2022 at 08:56
>
>
>
> To: users@cloudstack.apache.org<ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org> <us...@cloudstack.apache.org>
>
>
>
> Subject: Re: Recurring Snaps Failing
>
>
>
>
>
>
>
> Hi
>
>
>
>
>
>
>
> I ran a few more tests and can now finally make sense of the failure:
>
>
>
> 2022-10-17 13:39:08,556 DEBUG [o.a.c.s.d.d.CloudStackPrimaryDataStoreDriverImpl] (Work-Job-Executor-1:ctx-93cb161b job-806/job-807 ctx-3f9ccf45) (logid:8fd10b20) Failed to take snapshot: 23
>
>
>
> 2022-10-17 13:39:08,558 DEBUG [o.a.c.s.s.SnapshotServiceImpl] (Work-Job-Executor-1:ctx-93cb161b job-806/job-807 ctx-3f9ccf45) (logid:8fd10b20) create snapshot Test001 failed: com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:1, com.cloud.exception.OperationTimedoutException: Commands 7160441932542378205 to Host 1 timed out after 3600
>
>
>
>
>
>
>
> Seems to be a time out error. But there's no firewalls on the KVM host and NFS server to prevent connections and it worked fine before and randomly just stopped.
>
>
>
>
>
>
>
> So I still have no idea where to solve this as all other necessary settings are in place. Any ideas?
>
>
>
> On 10/17/22 08:51, Granwille Strauss wrote:
>
>
>
>
>
>
>
> Hi Guys
>
>
>
>
>
>
>
> I enabled recurring snap shots for my Linux VMs and they all seem to be failing with the following error:
>
>
>
> root@hostname ~ $ grep error /var/log/cloudstack/management/management-server.log
>
>
>
>
>
>
>
>
>
>
>
> 2022-10-17 03:01:29,474 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-27:ctx-99d7d702 job-796 ctx-f26a81fe) (logid:f9a61b00) Failed to create snapshot due to an internal error creating snapshot for volume 635fd5b5-bf14-4cf4-979c-be8a649d5189
>
>
>
> 2022-10-17 03:01:29,476 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-27:ctx-99d7d702 job-796) (logid:f9a61b00) Complete async job-796, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 635fd5b5-bf14-4cf4-979c-be8a649d5189"}
>
>
>
> 2022-10-17 03:01:29,478 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-26:ctx-c7865967 job-794 ctx-0700afb0) (logid:5541113c) Failed to create snapshot due to an internal error creating snapshot for volume 70ffef79-3aad-43ab-93d1-153fcd34ad15
>
>
>
> 2022-10-17 03:01:29,480 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-26:ctx-c7865967 job-794) (logid:5541113c) Complete async job-794, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 70ffef79-3aad-43ab-93d1-153fcd34ad15"}
>
>
>
>
>
>
>
> All I am getting out of this is internal error and nothing else. I have checked other logs and I am not quite sure where else to look.
>
>
>
>
>
>
>
> Any assistance in this regard will be appreciated.
>
>
>
> --
>
>
>
>
>
>
>
> Regards / Groete
>
>
>
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/>
>
>
>
> Granwille Strauss // Senior Systems Admin
>
>
>
>
>
>
>
> e: granwille@namhost.com<ma...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com>
>
>
>
> m: +264 81 323 1260<tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260>
>
>
>
> w: www.namhost.com<http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/>
>
>
>
>
>
>
>
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>
>
>
>
>
>
>
> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>
>
>
> Namhost Internet Services (Pty) Ltd,
>
>
>
>
>
>
> 24 Black Eagle Rd, Hermanus <https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>
>
>
>
>
>
>
> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go to https://www.namhost.com/privacy-policy
>
>
>
>
>
>
>
> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>
>
>
> --
>
>
>
>
>
>
>
> Regards / Groete
>
>
>
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/>
>
>
>
> Granwille Strauss // Senior Systems Admin
>
>
>
>
>
>
>
> e: granwille@namhost.com<ma...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com>
>
>
>
> m: +264 81 323 1260<tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260>
>
>
>
> w: www.namhost.com<http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/>
>
>
>
>
>
>
>
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>
>
>
>
>
>
>
> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>
>
>
> Namhost Internet Services (Pty) Ltd,
>
>
>
>
>
>
> 24 Black Eagle Rd, Hermanus <https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>
>
>
>
>
>
>
> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go to https://www.namhost.com/privacy-policy
>
>
>
>
>
>
>
> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> --
>
>
>
> Regards / Groete
>
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/>
>
> Granwille Strauss // Senior Systems Admin
>
>
>
> e: granwille@namhost.com<ma...@namhost.com> <gr...@namhost.com> <gr...@namhost.com> <gr...@namhost.com>
>
> m: +264 81 323 1260<tel:+264813231260><tel:+264813231260>
>
> w: www.namhost.com<http://www.namhost.com> <http://www.namhost.com><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/>
>
>
>
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>
>
>
> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>
> Namhost Internet Services (Pty) Ltd,
>
>
> 24 Black Eagle Rd, Hermanus <https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>
>
>
> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go to https://www.namhost.com/privacy-policy
>
>
> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>
>
>
>
>
>
>
>
> --
> Regards / Groete
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/> <https://www.namhost.com/>
> Granwille Strauss // Senior Systems Admin
>
> e: granwille@namhost.com<ma...@namhost.com> <gr...@namhost.com>
> m: +264 81 323 1260<tel:+264813231260>
> w: www.namhost.com<https://www.namhost.com/> <https://www.namhost.com/>
>
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost> <https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost> <https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>
> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
> Namhost Internet Services (Pty) Ltd,
> 24 Black Eagle Rd, Hermanus <https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>
> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go to https://www.namhost.com/privacy-policy
>
> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
> --
>
> Regards / Groete
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/> <https://www.namhost.com/>
> Granwille Strauss // Senior Systems Admin
>
> e: granwille@namhost.com<ma...@namhost.com> <gr...@namhost.com>
> m: +264 81 323 1260<tel:+264813231260>
> w: www.namhost.com<https://www.namhost.com/> <https://www.namhost.com/>
>
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost> <https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost> <https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>
> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
> Namhost Internet Services (Pty) Ltd,
> 24 Black Eagle Rd, Hermanus <https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>
> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and s
>
> --
> Regards / Groete
>
> <https://www.namhost.com> Granwille Strauss // Senior Systems Admin
>
> *e:* granwille@namhost.com
> *m:* +264 81 323 1260 <+264813231260>
> *w:* www.namhost.com
>
> <https://www.facebook.com/namhost> <https://twitter.com/namhost>
> <https://www.instagram.com/namhostinternetservices/>
> <https://www.linkedin.com/company/namhos>
> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>
>
> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>
> Namhost Internet Services (Pty) Ltd,
>
> 24 Black Eagle Rd, Hermanus <https://www.google.com/maps/search/24+Black+Eagle+Rd,+Hermanus?entry=gmail&source=g>, 7210, RSA
>
>
>
> The content of this message is confidential. If you have received it by
> mistake, please inform us by email reply and then delete the message. It is
> forbidden to copy, forward, or in any way reveal the contents of this
> message to anyone without our explicit consent. The integrity and security
> of this email cannot be guaranteed over the Internet. Therefore, the sender
> will not be held liable for any damage caused by the message. For our full
> privacy policy and disclaimers, please go to https://www.namhost.com/
> privacy-policy
>
> [image: Powered by AdSigner]
> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>
Re: Recurring Snaps Failing
Posted by Granwille Strauss <gr...@namhost.com.INVALID>.
The error seems to happen exactly at 1 hour intervals. All of them have
failed exactly at one hour. Also, these are all volume snapshots that I
am taking, so the logs I sent you so far are all in this regard. They
are not VM snaphots, nor are they powered off. I am running a new volume
snap shot again, this time VM powered off lets see.
I have also rebooted my entire KVM server and my management server. So
far zero luck.
On 10/17/22 20:26, Nicolas Vazquez wrote:
> Can you try taking a manual volume snapshot and share both mgmt and agent logs please?
>
> Regards,
> Nicolas Vazquez
>
>
> From: Granwille Strauss<gr...@namhost.com>
> Date: Monday, 17 October 2022 at 13:15
> To:users@cloudstack.apache.org <us...@cloudstack.apache.org>
> Cc: Nicolas Vazquez<Ni...@shapeblue.com>
> Subject: Re: Recurring Snaps Failing
>
> Hi
>
> I also see another issue with snaps, I deleted all the failed snaps via the cloud management UI. But to my surprise, they're all still at /mnt/primary/snapshots/ see attached screenshot. Something odd is going on with snapshots.
> On 10/17/22 17:35, Granwille Strauss wrote:
>
> Hi
>
> I am not 100% if I enabled debug correctly, I followed thishttps://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+agent+debug and ran
> sed -i 's/INFO/DEBUG/g' /etc/cloudstack/agent/log4j-cloud.xml
>
> Attached are the two log files. Same error. I reboot SSVM & vRouter prior no luck.
> On 10/17/22 16:00, Granwille Strauss wrote:
>
> Hi
>
> No KVM is on its own server node and management on a different server node. However, management is is primary and secondary storage via NFS. I will enable DEBUG and re-run manual snap to replicate the error.
> On 10/17/22 15:42, Nicolas Vazquez wrote:
> Thanks, are the KVM agent and management server running on the same host? Can you increase the log level on the KVM agent logs to DEBUG and retry?
>
> Regards,
> Nicolas Vazquez
>
>
>
>
>
>
> From: Granwille Strauss<gr...@namhost.com>
> Date: Monday, 17 October 2022 at 10:04
> To:users@cloudstack.apache.org<ma...@cloudstack.apache.org> <us...@cloudstack.apache.org>
> Cc: Nicolas Vazquez<Ni...@shapeblue.com>
> Subject: Re: Recurring Snaps Failing
>
> Please find attached TXT
> On 10/17/22 14:54, Nicolas Vazquez wrote:
>
> Ok, seems like the command is not reaching the KVM host, can you share the full management server logs for job-796?
>
>
>
> Regards,
>
> Nicolas Vazquez
>
>
>
>
>
> From: Granwille Strauss<gr...@namhost.com>
>
> Date: Monday, 17 October 2022 at 09:32
>
> To:users@cloudstack.apache.org<ma...@cloudstack.apache.org> <us...@cloudstack.apache.org>
>
> Cc: Nicolas Vazquez<Ni...@shapeblue.com>
>
> Subject: Re: Recurring Snaps Failing
>
>
>
> Hi
>
>
>
> This is all agent threw during the snap process nothing else:
>
> 2022-10-17 13:39:12,375 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:a864f9f6) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
>
> 2022-10-17 13:40:13,276 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:445216f3) Trying to fetch storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297 from libvirt
>
> 2022-10-17 13:40:13,289 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:445216f3) Asking libvirt to refresh storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297
>
> 2022-10-17 13:40:13,378 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:445216f3) Trying to fetch storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89 from libvirt
>
> 2022-10-17 13:40:13,393 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:445216f3) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
>
> 2022-10-17 13:41:13,950 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:87ecfee0) Trying to fetch storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297 from libvirt
>
> 2022-10-17 13:41:13,964 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:87ecfee0) Asking libvirt to refresh storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297
>
> 2022-10-17 13:41:14,054 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:87ecfee0) Trying to fetch storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89 from libvirt
>
> 2022-10-17 13:41:14,067 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:87ecfee0) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
>
> 2022-10-17 13:41:57,753 WARN [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-3:null) (logid:d91c35e5) Expected 1 answers while executing SetMonitorServiceCommand but received 3
>
> 2022-10-17 13:41:57,938 INFO [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:null) (logid:1549c9a1) Fetching health check result for SEVERIPADDRESS and executing fresh checks: false
>
>
>
> On 10/17/22 14:29, Nicolas Vazquez wrote:
>
>
>
> Hi Groete,
>
>
>
>
>
>
>
> Can you share the CloudStack agent logs from the KVM host?
>
>
>
>
>
>
>
> Regards,
>
>
>
> Nicolas Vazquez
>
>
>
>
>
>
>
>
>
>
>
> From: Granwille Strauss<gr...@namhost.com.INVALID>
>
>
>
> Date: Monday, 17 October 2022 at 08:56
>
>
>
> To:users@cloudstack.apache.org<ma...@cloudstack.apache.org> <us...@cloudstack.apache.org>
>
>
>
> Subject: Re: Recurring Snaps Failing
>
>
>
>
>
>
>
> Hi
>
>
>
>
>
>
>
> I ran a few more tests and can now finally make sense of the failure:
>
>
>
> 2022-10-17 13:39:08,556 DEBUG [o.a.c.s.d.d.CloudStackPrimaryDataStoreDriverImpl] (Work-Job-Executor-1:ctx-93cb161b job-806/job-807 ctx-3f9ccf45) (logid:8fd10b20) Failed to take snapshot: 23
>
>
>
> 2022-10-17 13:39:08,558 DEBUG [o.a.c.s.s.SnapshotServiceImpl] (Work-Job-Executor-1:ctx-93cb161b job-806/job-807 ctx-3f9ccf45) (logid:8fd10b20) create snapshot Test001 failed: com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:1, com.cloud.exception.OperationTimedoutException: Commands 7160441932542378205 to Host 1 timed out after 3600
>
>
>
>
>
>
>
> Seems to be a time out error. But there's no firewalls on the KVM host and NFS server to prevent connections and it worked fine before and randomly just stopped.
>
>
>
>
>
>
>
> So I still have no idea where to solve this as all other necessary settings are in place. Any ideas?
>
>
>
> On 10/17/22 08:51, Granwille Strauss wrote:
>
>
>
>
>
>
>
> Hi Guys
>
>
>
>
>
>
>
> I enabled recurring snap shots for my Linux VMs and they all seem to be failing with the following error:
>
>
>
> root@hostname ~ $ grep error /var/log/cloudstack/management/management-server.log
>
>
>
>
>
>
>
>
>
>
>
> 2022-10-17 03:01:29,474 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-27:ctx-99d7d702 job-796 ctx-f26a81fe) (logid:f9a61b00) Failed to create snapshot due to an internal error creating snapshot for volume 635fd5b5-bf14-4cf4-979c-be8a649d5189
>
>
>
> 2022-10-17 03:01:29,476 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-27:ctx-99d7d702 job-796) (logid:f9a61b00) Complete async job-796, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 635fd5b5-bf14-4cf4-979c-be8a649d5189"}
>
>
>
> 2022-10-17 03:01:29,478 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-26:ctx-c7865967 job-794 ctx-0700afb0) (logid:5541113c) Failed to create snapshot due to an internal error creating snapshot for volume 70ffef79-3aad-43ab-93d1-153fcd34ad15
>
>
>
> 2022-10-17 03:01:29,480 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-26:ctx-c7865967 job-794) (logid:5541113c) Complete async job-794, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 70ffef79-3aad-43ab-93d1-153fcd34ad15"}
>
>
>
>
>
>
>
> All I am getting out of this is internal error and nothing else. I have checked other logs and I am not quite sure where else to look.
>
>
>
>
>
>
>
> Any assistance in this regard will be appreciated.
>
>
>
> --
>
>
>
>
>
>
>
> Regards / Groete
>
>
>
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/>
>
>
>
> Granwille Strauss // Senior Systems Admin
>
>
>
>
>
>
>
> e:granwille@namhost.com<ma...@namhost.com>
>
>
>
> m: +264 81 323 1260<tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260>
>
>
>
> w:www.namhost.com<http://www.namhost.com><http://www.namhost.com><http://www.namhost.com><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/>
>
>
>
>
>
>
>
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>
>
>
>
>
>
>
> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>
>
>
> Namhost Internet Services (Pty) Ltd,
>
>
>
>
>
>
>
> 24 Black Eagle Rd, Hermanus, 7210, RSA
>
>
>
>
>
>
>
> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>
>
>
>
>
>
>
> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>
>
>
> --
>
>
>
>
>
>
>
> Regards / Groete
>
>
>
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/>
>
>
>
> Granwille Strauss // Senior Systems Admin
>
>
>
>
>
>
>
> e:granwille@namhost.com<ma...@namhost.com>
>
>
>
> m: +264 81 323 1260<tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260>
>
>
>
> w:www.namhost.com<http://www.namhost.com><http://www.namhost.com><http://www.namhost.com><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/>
>
>
>
>
>
>
>
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>
>
>
>
>
>
>
> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>
>
>
> Namhost Internet Services (Pty) Ltd,
>
>
>
>
>
>
>
> 24 Black Eagle Rd, Hermanus, 7210, RSA
>
>
>
>
>
>
>
> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>
>
>
>
>
>
>
> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> --
>
>
>
> Regards / Groete
>
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/><https://www.namhost.com/>
>
> Granwille Strauss // Senior Systems Admin
>
>
>
> e:granwille@namhost.com<ma...@namhost.com>
>
> m: +264 81 323 1260<tel:+264813231260><tel:+264813231260>
>
> w:www.namhost.com<http://www.namhost.com><https://www.namhost.com/><https://www.namhost.com/>
>
>
>
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost><https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost><https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>
>
>
> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>
> Namhost Internet Services (Pty) Ltd,
>
>
>
> 24 Black Eagle Rd, Hermanus, 7210, RSA
>
>
>
> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>
>
>
> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>
>
>
>
>
>
>
>
> --
> Regards / Groete
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
> Granwille Strauss // Senior Systems Admin
>
> e:granwille@namhost.com<ma...@namhost.com>
> m: +264 81 323 1260<tel:+264813231260>
> w:www.namhost.com<https://www.namhost.com/>
>
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>
> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
> Namhost Internet Services (Pty) Ltd,
>
> 24 Black Eagle Rd, Hermanus, 7210, RSA
>
> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>
> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
> --
>
> Regards / Groete
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
> Granwille Strauss // Senior Systems Admin
>
> e:granwille@namhost.com<ma...@namhost.com>
> m: +264 81 323 1260<tel:+264813231260>
> w:www.namhost.com<https://www.namhost.com/>
>
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>
> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
> Namhost Internet Services (Pty) Ltd,
>
> 24 Black Eagle Rd, Hermanus, 7210, RSA
>
> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>
> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
> --
>
> Regards / Groete
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
> Granwille Strauss // Senior Systems Admin
>
> e:granwille@namhost.com<ma...@namhost.com>
> m: +264 81 323 1260<tel:+264813231260>
> w:www.namhost.com<https://www.namhost.com/>
>
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>
> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
> Namhost Internet Services (Pty) Ltd,
>
> 24 Black Eagle Rd, Hermanus, 7210, RSA
>
> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>
> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
> --
>
> Regards / Groete
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
> Granwille Strauss // Senior Systems Admin
>
> e:granwille@namhost.com<ma...@namhost.com>
> m: +264 81 323 1260<tel:+264813231260>
> w:www.namhost.com<https://www.namhost.com/>
>
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>
> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
> Namhost Internet Services (Pty) Ltd,
>
> 24 Black Eagle Rd, Hermanus, 7210, RSA
>
> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>
> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>
>
>
>
--
Regards / Groete
<https://www.namhost.com> Granwille Strauss // Senior Systems Admin
*e:* granwille@namhost.com
*m:* +264 81 323 1260 <tel:+264813231260>
*w:* www.namhost.com <https://www.namhost.com/>
<https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
Namhost Internet Services (Pty) Ltd,
24 Black Eagle Rd, Hermanus, 7210, RSA
The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It
is forbidden to copy, forward, or in any way reveal the contents of this
message to anyone without our explicit consent. The integrity and
security of this email cannot be guaranteed over the Internet.
Therefore, the sender will not be held liable for any damage caused by
the message. For our full privacy policy and disclaimers, please go to
https://www.namhost.com/privacy-policy
Powered by AdSigner
<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
Re: Recurring Snaps Failing
Posted by Nicolas Vazquez <Ni...@shapeblue.com>.
Can you try taking a manual volume snapshot and share both mgmt and agent logs please?
Regards,
Nicolas Vazquez
From: Granwille Strauss <gr...@namhost.com>
Date: Monday, 17 October 2022 at 13:15
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Cc: Nicolas Vazquez <Ni...@shapeblue.com>
Subject: Re: Recurring Snaps Failing
Hi
I also see another issue with snaps, I deleted all the failed snaps via the cloud management UI. But to my surprise, they're all still at /mnt/primary/snapshots/ see attached screenshot. Something odd is going on with snapshots.
On 10/17/22 17:35, Granwille Strauss wrote:
Hi
I am not 100% if I enabled debug correctly, I followed this https://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+agent+debug and ran
sed -i 's/INFO/DEBUG/g' /etc/cloudstack/agent/log4j-cloud.xml
Attached are the two log files. Same error. I reboot SSVM & vRouter prior no luck.
On 10/17/22 16:00, Granwille Strauss wrote:
Hi
No KVM is on its own server node and management on a different server node. However, management is is primary and secondary storage via NFS. I will enable DEBUG and re-run manual snap to replicate the error.
On 10/17/22 15:42, Nicolas Vazquez wrote:
Thanks, are the KVM agent and management server running on the same host? Can you increase the log level on the KVM agent logs to DEBUG and retry?
Regards,
Nicolas Vazquez
From: Granwille Strauss <gr...@namhost.com>
Date: Monday, 17 October 2022 at 10:04
To: users@cloudstack.apache.org<ma...@cloudstack.apache.org> <us...@cloudstack.apache.org>
Cc: Nicolas Vazquez <Ni...@shapeblue.com>
Subject: Re: Recurring Snaps Failing
Please find attached TXT
On 10/17/22 14:54, Nicolas Vazquez wrote:
Ok, seems like the command is not reaching the KVM host, can you share the full management server logs for job-796?
Regards,
Nicolas Vazquez
From: Granwille Strauss <gr...@namhost.com>
Date: Monday, 17 October 2022 at 09:32
To: users@cloudstack.apache.org<ma...@cloudstack.apache.org> <us...@cloudstack.apache.org>
Cc: Nicolas Vazquez <Ni...@shapeblue.com>
Subject: Re: Recurring Snaps Failing
Hi
This is all agent threw during the snap process nothing else:
2022-10-17 13:39:12,375 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:a864f9f6) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
2022-10-17 13:40:13,276 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:445216f3) Trying to fetch storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297 from libvirt
2022-10-17 13:40:13,289 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:445216f3) Asking libvirt to refresh storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297
2022-10-17 13:40:13,378 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:445216f3) Trying to fetch storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89 from libvirt
2022-10-17 13:40:13,393 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:445216f3) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
2022-10-17 13:41:13,950 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:87ecfee0) Trying to fetch storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297 from libvirt
2022-10-17 13:41:13,964 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:87ecfee0) Asking libvirt to refresh storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297
2022-10-17 13:41:14,054 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:87ecfee0) Trying to fetch storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89 from libvirt
2022-10-17 13:41:14,067 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:87ecfee0) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
2022-10-17 13:41:57,753 WARN [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-3:null) (logid:d91c35e5) Expected 1 answers while executing SetMonitorServiceCommand but received 3
2022-10-17 13:41:57,938 INFO [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:null) (logid:1549c9a1) Fetching health check result for SEVERIPADDRESS and executing fresh checks: false
On 10/17/22 14:29, Nicolas Vazquez wrote:
Hi Groete,
Can you share the CloudStack agent logs from the KVM host?
Regards,
Nicolas Vazquez
From: Granwille Strauss <gr...@namhost.com.INVALID>
Date: Monday, 17 October 2022 at 08:56
To: users@cloudstack.apache.org<ma...@cloudstack.apache.org> <us...@cloudstack.apache.org>
Subject: Re: Recurring Snaps Failing
Hi
I ran a few more tests and can now finally make sense of the failure:
2022-10-17 13:39:08,556 DEBUG [o.a.c.s.d.d.CloudStackPrimaryDataStoreDriverImpl] (Work-Job-Executor-1:ctx-93cb161b job-806/job-807 ctx-3f9ccf45) (logid:8fd10b20) Failed to take snapshot: 23
2022-10-17 13:39:08,558 DEBUG [o.a.c.s.s.SnapshotServiceImpl] (Work-Job-Executor-1:ctx-93cb161b job-806/job-807 ctx-3f9ccf45) (logid:8fd10b20) create snapshot Test001 failed: com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:1, com.cloud.exception.OperationTimedoutException: Commands 7160441932542378205 to Host 1 timed out after 3600
Seems to be a time out error. But there's no firewalls on the KVM host and NFS server to prevent connections and it worked fine before and randomly just stopped.
So I still have no idea where to solve this as all other necessary settings are in place. Any ideas?
On 10/17/22 08:51, Granwille Strauss wrote:
Hi Guys
I enabled recurring snap shots for my Linux VMs and they all seem to be failing with the following error:
root@hostname ~ $ grep error /var/log/cloudstack/management/management-server.log
2022-10-17 03:01:29,474 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-27:ctx-99d7d702 job-796 ctx-f26a81fe) (logid:f9a61b00) Failed to create snapshot due to an internal error creating snapshot for volume 635fd5b5-bf14-4cf4-979c-be8a649d5189
2022-10-17 03:01:29,476 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-27:ctx-99d7d702 job-796) (logid:f9a61b00) Complete async job-796, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 635fd5b5-bf14-4cf4-979c-be8a649d5189"}
2022-10-17 03:01:29,478 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-26:ctx-c7865967 job-794 ctx-0700afb0) (logid:5541113c) Failed to create snapshot due to an internal error creating snapshot for volume 70ffef79-3aad-43ab-93d1-153fcd34ad15
2022-10-17 03:01:29,480 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-26:ctx-c7865967 job-794) (logid:5541113c) Complete async job-794, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 70ffef79-3aad-43ab-93d1-153fcd34ad15"}
All I am getting out of this is internal error and nothing else. I have checked other logs and I am not quite sure where else to look.
Any assistance in this regard will be appreciated.
--
Regards / Groete
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/>
Granwille Strauss // Senior Systems Admin
e: granwille@namhost.com<ma...@namhost.com>
m: +264 81 323 1260<tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260>
w: www.namhost.com<http://www.namhost.com><http://www.namhost.com><http://www.namhost.com><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/>
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
[https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
Namhost Internet Services (Pty) Ltd,
24 Black Eagle Rd, Hermanus, 7210, RSA
The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go to https://www.namhost.com/privacy-policy
[Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
--
Regards / Groete
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/>
Granwille Strauss // Senior Systems Admin
e: granwille@namhost.com<ma...@namhost.com>
m: +264 81 323 1260<tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260>
w: www.namhost.com<http://www.namhost.com><http://www.namhost.com><http://www.namhost.com><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/>
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
[https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
Namhost Internet Services (Pty) Ltd,
24 Black Eagle Rd, Hermanus, 7210, RSA
The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go to https://www.namhost.com/privacy-policy
[Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
--
Regards / Groete
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/><https://www.namhost.com/>
Granwille Strauss // Senior Systems Admin
e: granwille@namhost.com<ma...@namhost.com>
m: +264 81 323 1260<tel:+264813231260><tel:+264813231260>
w: www.namhost.com<http://www.namhost.com><https://www.namhost.com/><https://www.namhost.com/>
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost><https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost><https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
[https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
Namhost Internet Services (Pty) Ltd,
24 Black Eagle Rd, Hermanus, 7210, RSA
The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go to https://www.namhost.com/privacy-policy
<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
--
Regards / Groete
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
Granwille Strauss // Senior Systems Admin
e: granwille@namhost.com<ma...@namhost.com>
m: +264 81 323 1260<tel:+264813231260>
w: www.namhost.com<https://www.namhost.com/>
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
[https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
Namhost Internet Services (Pty) Ltd,
24 Black Eagle Rd, Hermanus, 7210, RSA
The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go to https://www.namhost.com/privacy-policy
[Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
--
Regards / Groete
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
Granwille Strauss // Senior Systems Admin
e: granwille@namhost.com<ma...@namhost.com>
m: +264 81 323 1260<tel:+264813231260>
w: www.namhost.com<https://www.namhost.com/>
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
[https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
Namhost Internet Services (Pty) Ltd,
24 Black Eagle Rd, Hermanus, 7210, RSA
The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go to https://www.namhost.com/privacy-policy
[Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
--
Regards / Groete
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
Granwille Strauss // Senior Systems Admin
e: granwille@namhost.com<ma...@namhost.com>
m: +264 81 323 1260<tel:+264813231260>
w: www.namhost.com<https://www.namhost.com/>
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
[https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
Namhost Internet Services (Pty) Ltd,
24 Black Eagle Rd, Hermanus, 7210, RSA
The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go to https://www.namhost.com/privacy-policy
[Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
--
Regards / Groete
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
Granwille Strauss // Senior Systems Admin
e: granwille@namhost.com<ma...@namhost.com>
m: +264 81 323 1260<tel:+264813231260>
w: www.namhost.com<https://www.namhost.com/>
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
[https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
Namhost Internet Services (Pty) Ltd,
24 Black Eagle Rd, Hermanus, 7210, RSA
The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go to https://www.namhost.com/privacy-policy
[Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
Re: Recurring Snaps Failing
Posted by Granwille Strauss <gr...@namhost.com.INVALID>.
Hi
I also see another issue with snaps, I deleted all the failed snaps via
the cloud management UI. But to my surprise, they're all still at
/mnt/primary/snapshots/ see attached screenshot. Something odd is going
on with snapshots.
On 10/17/22 17:35, Granwille Strauss wrote:
>
> Hi
>
> I am not 100% if I enabled debug correctly, I followed this
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+agent+debug
> and ran
>
>> |sed -i||'s/INFO/DEBUG/g'||/etc/cloudstack/agent/log4j-cloud.xml|
>
> |Attached are the two log files. Same error. I reboot SSVM & vRouter
> prior no luck. |
>
> On 10/17/22 16:00, Granwille Strauss wrote:
>>
>> Hi
>>
>> No KVM is on its own server node and management on a different server
>> node. However, management is is primary and secondary storage via
>> NFS. I will enable DEBUG and re-run manual snap to replicate the error.
>>
>> On 10/17/22 15:42, Nicolas Vazquez wrote:
>>>
>>> Thanks, are the KVM agent and management server running on the same
>>> host? Can you increase the log level on the KVM agent logs to DEBUG
>>> and retry?
>>>
>>> Regards,
>>>
>>> Nicolas Vazquez
>>>
>>>
>>> *From: *Granwille Strauss <gr...@namhost.com>
>>> *Date: *Monday, 17 October 2022 at 10:04
>>> *To: *users@cloudstack.apache.org <us...@cloudstack.apache.org>
>>> *Cc: *Nicolas Vazquez <Ni...@shapeblue.com>
>>> *Subject: *Re: Recurring Snaps Failing
>>>
>>> Please find attached TXT
>>>
>>> On 10/17/22 14:54, Nicolas Vazquez wrote:
>>>
>>> Ok, seems like the command is not reaching the KVM host, can you share the full management server logs for job-796?
>>>
>>> Regards,
>>>
>>> Nicolas Vazquez
>>>
>>> From: Granwille Strauss<gr...@namhost.com> <ma...@namhost.com>
>>>
>>> Date: Monday, 17 October 2022 at 09:32
>>>
>>> To:users@cloudstack.apache.org <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org>
>>>
>>> Cc: Nicolas Vazquez<Ni...@shapeblue.com> <ma...@shapeblue.com>
>>>
>>> Subject: Re: Recurring Snaps Failing
>>>
>>> Hi
>>>
>>> This is all agent threw during the snap process nothing else:
>>>
>>> 2022-10-17 13:39:12,375 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:a864f9f6) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
>>>
>>> 2022-10-17 13:40:13,276 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:445216f3) Trying to fetch storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297 from libvirt
>>>
>>> 2022-10-17 13:40:13,289 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:445216f3) Asking libvirt to refresh storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297
>>>
>>> 2022-10-17 13:40:13,378 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:445216f3) Trying to fetch storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89 from libvirt
>>>
>>> 2022-10-17 13:40:13,393 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:445216f3) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
>>>
>>> 2022-10-17 13:41:13,950 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:87ecfee0) Trying to fetch storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297 from libvirt
>>>
>>> 2022-10-17 13:41:13,964 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:87ecfee0) Asking libvirt to refresh storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297
>>>
>>> 2022-10-17 13:41:14,054 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:87ecfee0) Trying to fetch storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89 from libvirt
>>>
>>> 2022-10-17 13:41:14,067 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:87ecfee0) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
>>>
>>> 2022-10-17 13:41:57,753 WARN [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-3:null) (logid:d91c35e5) Expected 1 answers while executing SetMonitorServiceCommand but received 3
>>>
>>> 2022-10-17 13:41:57,938 INFO [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:null) (logid:1549c9a1) Fetching health check result for SEVERIPADDRESS and executing fresh checks: false
>>>
>>> On 10/17/22 14:29, Nicolas Vazquez wrote:
>>>
>>> Hi Groete,
>>>
>>> Can you share the CloudStack agent logs from the KVM host?
>>>
>>> Regards,
>>>
>>> Nicolas Vazquez
>>>
>>> From: Granwille Strauss<gr...@namhost.com.INVALID> <ma...@namhost.com.INVALID> <ma...@namhost.com.INVALID>
>>>
>>> Date: Monday, 17 October 2022 at 08:56
>>>
>>> To:users@cloudstack.apache.org<ma...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <ma...@cloudstack.apache.org>
>>>
>>> Subject: Re: Recurring Snaps Failing
>>>
>>> Hi
>>>
>>> I ran a few more tests and can now finally make sense of the failure:
>>>
>>> 2022-10-17 13:39:08,556 DEBUG [o.a.c.s.d.d.CloudStackPrimaryDataStoreDriverImpl] (Work-Job-Executor-1:ctx-93cb161b job-806/job-807 ctx-3f9ccf45) (logid:8fd10b20) Failed to take snapshot: 23
>>>
>>> 2022-10-17 13:39:08,558 DEBUG [o.a.c.s.s.SnapshotServiceImpl] (Work-Job-Executor-1:ctx-93cb161b job-806/job-807 ctx-3f9ccf45) (logid:8fd10b20) create snapshot Test001 failed: com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:1, com.cloud.exception.OperationTimedoutException: Commands 7160441932542378205 to Host 1 timed out after 3600
>>>
>>> Seems to be a time out error. But there's no firewalls on the KVM host and NFS server to prevent connections and it worked fine before and randomly just stopped.
>>>
>>> So I still have no idea where to solve this as all other necessary settings are in place. Any ideas?
>>>
>>> On 10/17/22 08:51, Granwille Strauss wrote:
>>>
>>> Hi Guys
>>>
>>> I enabled recurring snap shots for my Linux VMs and they all seem to be failing with the following error:
>>>
>>> root@hostname ~ $ grep error /var/log/cloudstack/management/management-server.log
>>>
>>> 2022-10-17 03:01:29,474 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-27:ctx-99d7d702 job-796 ctx-f26a81fe) (logid:f9a61b00) Failed to create snapshot due to an internal error creating snapshot for volume 635fd5b5-bf14-4cf4-979c-be8a649d5189
>>>
>>> 2022-10-17 03:01:29,476 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-27:ctx-99d7d702 job-796) (logid:f9a61b00) Complete async job-796, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 635fd5b5-bf14-4cf4-979c-be8a649d5189"}
>>>
>>> 2022-10-17 03:01:29,478 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-26:ctx-c7865967 job-794 ctx-0700afb0) (logid:5541113c) Failed to create snapshot due to an internal error creating snapshot for volume 70ffef79-3aad-43ab-93d1-153fcd34ad15
>>>
>>> 2022-10-17 03:01:29,480 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-26:ctx-c7865967 job-794) (logid:5541113c) Complete async job-794, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 70ffef79-3aad-43ab-93d1-153fcd34ad15"}
>>>
>>> All I am getting out of this is internal error and nothing else. I have checked other logs and I am not quite sure where else to look.
>>>
>>> Any assistance in this regard will be appreciated.
>>>
>>> --
>>>
>>> Regards / Groete
>>>
>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/>
>>>
>>> Granwille Strauss // Senior Systems Admin
>>>
>>> e:granwille@namhost.com<ma...@namhost.com> <ma...@namhost.com> <ma...@namhost.com> <ma...@namhost.com>
>>>
>>> m: +264 81 323 1260<tel:+264813231260> <tel:+264813231260><tel:+264813231260> <tel:+264813231260>
>>>
>>> w:www.namhost.com <http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/>
>>>
>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>
>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>
>>> Namhost Internet Services (Pty) Ltd,
>>>
>>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>>
>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>
>>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>
>>> --
>>>
>>> Regards / Groete
>>>
>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/>
>>>
>>> Granwille Strauss // Senior Systems Admin
>>>
>>> e:granwille@namhost.com<ma...@namhost.com> <ma...@namhost.com> <ma...@namhost.com> <ma...@namhost.com>
>>>
>>> m: +264 81 323 1260<tel:+264813231260> <tel:+264813231260><tel:+264813231260> <tel:+264813231260>
>>>
>>> w:www.namhost.com <http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/>
>>>
>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>
>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>
>>> Namhost Internet Services (Pty) Ltd,
>>>
>>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>>
>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>
>>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>
>>> --
>>>
>>> Regards / Groete
>>>
>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/> <https://www.namhost.com/>
>>>
>>> Granwille Strauss // Senior Systems Admin
>>>
>>> e:granwille@namhost.com<ma...@namhost.com> <ma...@namhost.com>
>>>
>>> m: +264 81 323 1260<tel:+264813231260> <tel:+264813231260>
>>>
>>> w:www.namhost.com <http://www.namhost.com><https://www.namhost.com/> <https://www.namhost.com/>
>>>
>>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost> <https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost> <https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>
>>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>
>>> Namhost Internet Services (Pty) Ltd,
>>>
>>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>>
>>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>>
>>> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>
>>>
>>>
>>> --
>>>
>>> Regards / Groete
>>>
>>> <https://www.namhost.com/>
>>>
>>>
>>>
>>> Granwille Strauss // Senior Systems Admin
>>>
>>> *e:* granwille@namhost.com <ma...@namhost.com>
>>> *m:* +264 81 323 1260 <tel:+264813231260>
>>> *w:* www.namhost.com <https://www.namhost.com/>
>>>
>>> <https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>>
>>>
>>> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>>
>>> Namhost Internet Services (Pty) Ltd,
>>>
>>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>>
>>> The content of this message is confidential. If you have received it
>>> by mistake, please inform us by email reply and then delete the
>>> message. It is forbidden to copy, forward, or in any way reveal the
>>> contents of this message to anyone without our explicit consent. The
>>> integrity and security of this email cannot be guaranteed over the
>>> Internet. Therefore, the sender will not be held liable for any
>>> damage caused by the message. For our full privacy policy and
>>> disclaimers, please go to https://www.namhost.com/privacy-policy
>>>
>>> Powered by AdSigner
>>> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>>
>> --
>> Regards / Groete
>>
>> <https://www.namhost.com> Granwille Strauss // Senior Systems Admin
>>
>> *e:* granwille@namhost.com
>> *m:* +264 81 323 1260 <tel:+264813231260>
>> *w:* www.namhost.com <https://www.namhost.com/>
>>
>> <https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>
>> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>
>> Namhost Internet Services (Pty) Ltd,
>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>
>>
>> The content of this message is confidential. If you have received it
>> by mistake, please inform us by email reply and then delete the
>> message. It is forbidden to copy, forward, or in any way reveal the
>> contents of this message to anyone without our explicit consent. The
>> integrity and security of this email cannot be guaranteed over the
>> Internet. Therefore, the sender will not be held liable for any
>> damage caused by the message. For our full privacy policy and
>> disclaimers, please go to https://www.namhost.com/privacy-policy
>>
>> Powered by AdSigner
>> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
> --
> Regards / Groete
>
> <https://www.namhost.com> Granwille Strauss // Senior Systems Admin
>
> *e:* granwille@namhost.com
> *m:* +264 81 323 1260 <tel:+264813231260>
> *w:* www.namhost.com <https://www.namhost.com/>
>
> <https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>
> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>
> Namhost Internet Services (Pty) Ltd,
> 24 Black Eagle Rd, Hermanus, 7210, RSA
>
>
> The content of this message is confidential. If you have received it
> by mistake, please inform us by email reply and then delete the
> message. It is forbidden to copy, forward, or in any way reveal the
> contents of this message to anyone without our explicit consent. The
> integrity and security of this email cannot be guaranteed over the
> Internet. Therefore, the sender will not be held liable for any damage
> caused by the message. For our full privacy policy and disclaimers,
> please go to https://www.namhost.com/privacy-policy
>
> Powered by AdSigner
> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
--
Regards / Groete
<https://www.namhost.com> Granwille Strauss // Senior Systems Admin
*e:* granwille@namhost.com
*m:* +264 81 323 1260 <tel:+264813231260>
*w:* www.namhost.com <https://www.namhost.com/>
<https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
Namhost Internet Services (Pty) Ltd,
24 Black Eagle Rd, Hermanus, 7210, RSA
The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It
is forbidden to copy, forward, or in any way reveal the contents of this
message to anyone without our explicit consent. The integrity and
security of this email cannot be guaranteed over the Internet.
Therefore, the sender will not be held liable for any damage caused by
the message. For our full privacy policy and disclaimers, please go to
https://www.namhost.com/privacy-policy
Powered by AdSigner
<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
Re: Recurring Snaps Failing
Posted by Granwille Strauss <gr...@namhost.com.INVALID>.
Hi
I am not 100% if I enabled debug correctly, I followed this
https://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+agent+debug
and ran
> |sed -i||'s/INFO/DEBUG/g'||/etc/cloudstack/agent/log4j-cloud.xml|
|Attached are the two log files. Same error. I reboot SSVM & vRouter
prior no luck. |
On 10/17/22 16:00, Granwille Strauss wrote:
>
> Hi
>
> No KVM is on its own server node and management on a different server
> node. However, management is is primary and secondary storage via NFS.
> I will enable DEBUG and re-run manual snap to replicate the error.
>
> On 10/17/22 15:42, Nicolas Vazquez wrote:
>>
>> Thanks, are the KVM agent and management server running on the same
>> host? Can you increase the log level on the KVM agent logs to DEBUG
>> and retry?
>>
>> Regards,
>>
>> Nicolas Vazquez
>>
>>
>> *From: *Granwille Strauss <gr...@namhost.com>
>> *Date: *Monday, 17 October 2022 at 10:04
>> *To: *users@cloudstack.apache.org <us...@cloudstack.apache.org>
>> *Cc: *Nicolas Vazquez <Ni...@shapeblue.com>
>> *Subject: *Re: Recurring Snaps Failing
>>
>> Please find attached TXT
>>
>> On 10/17/22 14:54, Nicolas Vazquez wrote:
>>
>> Ok, seems like the command is not reaching the KVM host, can you share the full management server logs for job-796?
>>
>> Regards,
>>
>> Nicolas Vazquez
>>
>> From: Granwille Strauss<gr...@namhost.com> <ma...@namhost.com>
>>
>> Date: Monday, 17 October 2022 at 09:32
>>
>> To:users@cloudstack.apache.org <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org>
>>
>> Cc: Nicolas Vazquez<Ni...@shapeblue.com> <ma...@shapeblue.com>
>>
>> Subject: Re: Recurring Snaps Failing
>>
>> Hi
>>
>> This is all agent threw during the snap process nothing else:
>>
>> 2022-10-17 13:39:12,375 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:a864f9f6) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
>>
>> 2022-10-17 13:40:13,276 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:445216f3) Trying to fetch storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297 from libvirt
>>
>> 2022-10-17 13:40:13,289 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:445216f3) Asking libvirt to refresh storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297
>>
>> 2022-10-17 13:40:13,378 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:445216f3) Trying to fetch storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89 from libvirt
>>
>> 2022-10-17 13:40:13,393 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:445216f3) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
>>
>> 2022-10-17 13:41:13,950 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:87ecfee0) Trying to fetch storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297 from libvirt
>>
>> 2022-10-17 13:41:13,964 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:87ecfee0) Asking libvirt to refresh storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297
>>
>> 2022-10-17 13:41:14,054 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:87ecfee0) Trying to fetch storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89 from libvirt
>>
>> 2022-10-17 13:41:14,067 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:87ecfee0) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
>>
>> 2022-10-17 13:41:57,753 WARN [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-3:null) (logid:d91c35e5) Expected 1 answers while executing SetMonitorServiceCommand but received 3
>>
>> 2022-10-17 13:41:57,938 INFO [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:null) (logid:1549c9a1) Fetching health check result for SEVERIPADDRESS and executing fresh checks: false
>>
>> On 10/17/22 14:29, Nicolas Vazquez wrote:
>>
>> Hi Groete,
>>
>> Can you share the CloudStack agent logs from the KVM host?
>>
>> Regards,
>>
>> Nicolas Vazquez
>>
>> From: Granwille Strauss<gr...@namhost.com.INVALID> <ma...@namhost.com.INVALID> <ma...@namhost.com.INVALID>
>>
>> Date: Monday, 17 October 2022 at 08:56
>>
>> To:users@cloudstack.apache.org<ma...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <ma...@cloudstack.apache.org>
>>
>> Subject: Re: Recurring Snaps Failing
>>
>> Hi
>>
>> I ran a few more tests and can now finally make sense of the failure:
>>
>> 2022-10-17 13:39:08,556 DEBUG [o.a.c.s.d.d.CloudStackPrimaryDataStoreDriverImpl] (Work-Job-Executor-1:ctx-93cb161b job-806/job-807 ctx-3f9ccf45) (logid:8fd10b20) Failed to take snapshot: 23
>>
>> 2022-10-17 13:39:08,558 DEBUG [o.a.c.s.s.SnapshotServiceImpl] (Work-Job-Executor-1:ctx-93cb161b job-806/job-807 ctx-3f9ccf45) (logid:8fd10b20) create snapshot Test001 failed: com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:1, com.cloud.exception.OperationTimedoutException: Commands 7160441932542378205 to Host 1 timed out after 3600
>>
>> Seems to be a time out error. But there's no firewalls on the KVM host and NFS server to prevent connections and it worked fine before and randomly just stopped.
>>
>> So I still have no idea where to solve this as all other necessary settings are in place. Any ideas?
>>
>> On 10/17/22 08:51, Granwille Strauss wrote:
>>
>> Hi Guys
>>
>> I enabled recurring snap shots for my Linux VMs and they all seem to be failing with the following error:
>>
>> root@hostname ~ $ grep error /var/log/cloudstack/management/management-server.log
>>
>> 2022-10-17 03:01:29,474 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-27:ctx-99d7d702 job-796 ctx-f26a81fe) (logid:f9a61b00) Failed to create snapshot due to an internal error creating snapshot for volume 635fd5b5-bf14-4cf4-979c-be8a649d5189
>>
>> 2022-10-17 03:01:29,476 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-27:ctx-99d7d702 job-796) (logid:f9a61b00) Complete async job-796, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 635fd5b5-bf14-4cf4-979c-be8a649d5189"}
>>
>> 2022-10-17 03:01:29,478 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-26:ctx-c7865967 job-794 ctx-0700afb0) (logid:5541113c) Failed to create snapshot due to an internal error creating snapshot for volume 70ffef79-3aad-43ab-93d1-153fcd34ad15
>>
>> 2022-10-17 03:01:29,480 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-26:ctx-c7865967 job-794) (logid:5541113c) Complete async job-794, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 70ffef79-3aad-43ab-93d1-153fcd34ad15"}
>>
>> All I am getting out of this is internal error and nothing else. I have checked other logs and I am not quite sure where else to look.
>>
>> Any assistance in this regard will be appreciated.
>>
>> --
>>
>> Regards / Groete
>>
>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/>
>>
>> Granwille Strauss // Senior Systems Admin
>>
>> e:granwille@namhost.com<ma...@namhost.com> <ma...@namhost.com> <ma...@namhost.com> <ma...@namhost.com>
>>
>> m: +264 81 323 1260<tel:+264813231260> <tel:+264813231260><tel:+264813231260> <tel:+264813231260>
>>
>> w:www.namhost.com <http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/>
>>
>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>
>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>
>> Namhost Internet Services (Pty) Ltd,
>>
>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>
>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>
>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>
>> --
>>
>> Regards / Groete
>>
>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/>
>>
>> Granwille Strauss // Senior Systems Admin
>>
>> e:granwille@namhost.com<ma...@namhost.com> <ma...@namhost.com> <ma...@namhost.com> <ma...@namhost.com>
>>
>> m: +264 81 323 1260<tel:+264813231260> <tel:+264813231260><tel:+264813231260> <tel:+264813231260>
>>
>> w:www.namhost.com <http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/>
>>
>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>
>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>
>> Namhost Internet Services (Pty) Ltd,
>>
>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>
>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>
>> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>
>> --
>>
>> Regards / Groete
>>
>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/> <https://www.namhost.com/>
>>
>> Granwille Strauss // Senior Systems Admin
>>
>> e:granwille@namhost.com<ma...@namhost.com> <ma...@namhost.com>
>>
>> m: +264 81 323 1260<tel:+264813231260> <tel:+264813231260>
>>
>> w:www.namhost.com <http://www.namhost.com><https://www.namhost.com/> <https://www.namhost.com/>
>>
>> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost> <https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost> <https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>
>> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>
>> Namhost Internet Services (Pty) Ltd,
>>
>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>
>> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>>
>> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>
>>
>>
>> --
>>
>> Regards / Groete
>>
>> <https://www.namhost.com/>
>>
>>
>>
>> Granwille Strauss // Senior Systems Admin
>>
>> *e:* granwille@namhost.com <ma...@namhost.com>
>> *m:* +264 81 323 1260 <tel:+264813231260>
>> *w:* www.namhost.com <https://www.namhost.com/>
>>
>> <https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>>
>>
>> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>>
>> Namhost Internet Services (Pty) Ltd,
>>
>> 24 Black Eagle Rd, Hermanus, 7210, RSA
>>
>> The content of this message is confidential. If you have received it
>> by mistake, please inform us by email reply and then delete the
>> message. It is forbidden to copy, forward, or in any way reveal the
>> contents of this message to anyone without our explicit consent. The
>> integrity and security of this email cannot be guaranteed over the
>> Internet. Therefore, the sender will not be held liable for any
>> damage caused by the message. For our full privacy policy and
>> disclaimers, please go to https://www.namhost.com/privacy-policy
>>
>> Powered by AdSigner
>> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>>
> --
> Regards / Groete
>
> <https://www.namhost.com> Granwille Strauss // Senior Systems Admin
>
> *e:* granwille@namhost.com
> *m:* +264 81 323 1260 <tel:+264813231260>
> *w:* www.namhost.com <https://www.namhost.com/>
>
> <https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>
> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>
> Namhost Internet Services (Pty) Ltd,
> 24 Black Eagle Rd, Hermanus, 7210, RSA
>
>
> The content of this message is confidential. If you have received it
> by mistake, please inform us by email reply and then delete the
> message. It is forbidden to copy, forward, or in any way reveal the
> contents of this message to anyone without our explicit consent. The
> integrity and security of this email cannot be guaranteed over the
> Internet. Therefore, the sender will not be held liable for any damage
> caused by the message. For our full privacy policy and disclaimers,
> please go to https://www.namhost.com/privacy-policy
>
> Powered by AdSigner
> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
--
Regards / Groete
<https://www.namhost.com> Granwille Strauss // Senior Systems Admin
*e:* granwille@namhost.com
*m:* +264 81 323 1260 <tel:+264813231260>
*w:* www.namhost.com <https://www.namhost.com/>
<https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
Namhost Internet Services (Pty) Ltd,
24 Black Eagle Rd, Hermanus, 7210, RSA
The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It
is forbidden to copy, forward, or in any way reveal the contents of this
message to anyone without our explicit consent. The integrity and
security of this email cannot be guaranteed over the Internet.
Therefore, the sender will not be held liable for any damage caused by
the message. For our full privacy policy and disclaimers, please go to
https://www.namhost.com/privacy-policy
Powered by AdSigner
<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
Re: Recurring Snaps Failing
Posted by Granwille Strauss <gr...@namhost.com.INVALID>.
Hi
No KVM is on its own server node and management on a different server
node. However, management is is primary and secondary storage via NFS. I
will enable DEBUG and re-run manual snap to replicate the error.
On 10/17/22 15:42, Nicolas Vazquez wrote:
>
> Thanks, are the KVM agent and management server running on the same
> host? Can you increase the log level on the KVM agent logs to DEBUG
> and retry?
>
> Regards,
>
> Nicolas Vazquez
>
>
> *From: *Granwille Strauss <gr...@namhost.com>
> *Date: *Monday, 17 October 2022 at 10:04
> *To: *users@cloudstack.apache.org <us...@cloudstack.apache.org>
> *Cc: *Nicolas Vazquez <Ni...@shapeblue.com>
> *Subject: *Re: Recurring Snaps Failing
>
> Please find attached TXT
>
> On 10/17/22 14:54, Nicolas Vazquez wrote:
>
> Ok, seems like the command is not reaching the KVM host, can you share the full management server logs for job-796?
>
> Regards,
>
> Nicolas Vazquez
>
> From: Granwille Strauss<gr...@namhost.com> <ma...@namhost.com>
>
> Date: Monday, 17 October 2022 at 09:32
>
> To:users@cloudstack.apache.org <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org>
>
> Cc: Nicolas Vazquez<Ni...@shapeblue.com> <ma...@shapeblue.com>
>
> Subject: Re: Recurring Snaps Failing
>
> Hi
>
> This is all agent threw during the snap process nothing else:
>
> 2022-10-17 13:39:12,375 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:a864f9f6) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
>
> 2022-10-17 13:40:13,276 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:445216f3) Trying to fetch storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297 from libvirt
>
> 2022-10-17 13:40:13,289 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:445216f3) Asking libvirt to refresh storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297
>
> 2022-10-17 13:40:13,378 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:445216f3) Trying to fetch storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89 from libvirt
>
> 2022-10-17 13:40:13,393 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:445216f3) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
>
> 2022-10-17 13:41:13,950 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:87ecfee0) Trying to fetch storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297 from libvirt
>
> 2022-10-17 13:41:13,964 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:87ecfee0) Asking libvirt to refresh storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297
>
> 2022-10-17 13:41:14,054 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:87ecfee0) Trying to fetch storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89 from libvirt
>
> 2022-10-17 13:41:14,067 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:87ecfee0) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
>
> 2022-10-17 13:41:57,753 WARN [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-3:null) (logid:d91c35e5) Expected 1 answers while executing SetMonitorServiceCommand but received 3
>
> 2022-10-17 13:41:57,938 INFO [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:null) (logid:1549c9a1) Fetching health check result for SEVERIPADDRESS and executing fresh checks: false
>
> On 10/17/22 14:29, Nicolas Vazquez wrote:
>
> Hi Groete,
>
> Can you share the CloudStack agent logs from the KVM host?
>
> Regards,
>
> Nicolas Vazquez
>
> From: Granwille Strauss<gr...@namhost.com.INVALID> <ma...@namhost.com.INVALID> <ma...@namhost.com.INVALID>
>
> Date: Monday, 17 October 2022 at 08:56
>
> To:users@cloudstack.apache.org<ma...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <us...@cloudstack.apache.org> <ma...@cloudstack.apache.org> <ma...@cloudstack.apache.org>
>
> Subject: Re: Recurring Snaps Failing
>
> Hi
>
> I ran a few more tests and can now finally make sense of the failure:
>
> 2022-10-17 13:39:08,556 DEBUG [o.a.c.s.d.d.CloudStackPrimaryDataStoreDriverImpl] (Work-Job-Executor-1:ctx-93cb161b job-806/job-807 ctx-3f9ccf45) (logid:8fd10b20) Failed to take snapshot: 23
>
> 2022-10-17 13:39:08,558 DEBUG [o.a.c.s.s.SnapshotServiceImpl] (Work-Job-Executor-1:ctx-93cb161b job-806/job-807 ctx-3f9ccf45) (logid:8fd10b20) create snapshot Test001 failed: com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:1, com.cloud.exception.OperationTimedoutException: Commands 7160441932542378205 to Host 1 timed out after 3600
>
> Seems to be a time out error. But there's no firewalls on the KVM host and NFS server to prevent connections and it worked fine before and randomly just stopped.
>
> So I still have no idea where to solve this as all other necessary settings are in place. Any ideas?
>
> On 10/17/22 08:51, Granwille Strauss wrote:
>
> Hi Guys
>
> I enabled recurring snap shots for my Linux VMs and they all seem to be failing with the following error:
>
> root@hostname ~ $ grep error /var/log/cloudstack/management/management-server.log
>
> 2022-10-17 03:01:29,474 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-27:ctx-99d7d702 job-796 ctx-f26a81fe) (logid:f9a61b00) Failed to create snapshot due to an internal error creating snapshot for volume 635fd5b5-bf14-4cf4-979c-be8a649d5189
>
> 2022-10-17 03:01:29,476 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-27:ctx-99d7d702 job-796) (logid:f9a61b00) Complete async job-796, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 635fd5b5-bf14-4cf4-979c-be8a649d5189"}
>
> 2022-10-17 03:01:29,478 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-26:ctx-c7865967 job-794 ctx-0700afb0) (logid:5541113c) Failed to create snapshot due to an internal error creating snapshot for volume 70ffef79-3aad-43ab-93d1-153fcd34ad15
>
> 2022-10-17 03:01:29,480 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-26:ctx-c7865967 job-794) (logid:5541113c) Complete async job-794, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 70ffef79-3aad-43ab-93d1-153fcd34ad15"}
>
> All I am getting out of this is internal error and nothing else. I have checked other logs and I am not quite sure where else to look.
>
> Any assistance in this regard will be appreciated.
>
> --
>
> Regards / Groete
>
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/>
>
> Granwille Strauss // Senior Systems Admin
>
> e:granwille@namhost.com<ma...@namhost.com> <ma...@namhost.com> <ma...@namhost.com> <ma...@namhost.com>
>
> m: +264 81 323 1260<tel:+264813231260> <tel:+264813231260><tel:+264813231260> <tel:+264813231260>
>
> w:www.namhost.com <http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/>
>
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>
> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>
> Namhost Internet Services (Pty) Ltd,
>
> 24 Black Eagle Rd, Hermanus, 7210, RSA
>
> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>
> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>
> --
>
> Regards / Groete
>
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/>
>
> Granwille Strauss // Senior Systems Admin
>
> e:granwille@namhost.com<ma...@namhost.com> <ma...@namhost.com> <ma...@namhost.com> <ma...@namhost.com>
>
> m: +264 81 323 1260<tel:+264813231260> <tel:+264813231260><tel:+264813231260> <tel:+264813231260>
>
> w:www.namhost.com <http://www.namhost.com><http://www.namhost.com> <http://www.namhost.com><https://www.namhost.com/> <https://www.namhost.com/><https://www.namhost.com/> <https://www.namhost.com/>
>
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost> <https://www.facebook.com/namhost><https://www.facebook.com/namhost> <https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost> <https://twitter.com/namhost><https://twitter.com/namhost> <https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>
> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>
> Namhost Internet Services (Pty) Ltd,
>
> 24 Black Eagle Rd, Hermanus, 7210, RSA
>
> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>
> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>
> --
>
> Regards / Groete
>
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/> <https://www.namhost.com/>
>
> Granwille Strauss // Senior Systems Admin
>
> e:granwille@namhost.com<ma...@namhost.com> <ma...@namhost.com>
>
> m: +264 81 323 1260<tel:+264813231260> <tel:+264813231260>
>
> w:www.namhost.com <http://www.namhost.com><https://www.namhost.com/> <https://www.namhost.com/>
>
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost> <https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost> <https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/> <https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos> <https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>
> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>
> Namhost Internet Services (Pty) Ltd,
>
> 24 Black Eagle Rd, Hermanus, 7210, RSA
>
> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>
> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>
>
>
> --
>
> Regards / Groete
>
> <https://www.namhost.com/>
>
>
>
> Granwille Strauss // Senior Systems Admin
>
> *e:* granwille@namhost.com <ma...@namhost.com>
> *m:* +264 81 323 1260 <tel:+264813231260>
> *w:* www.namhost.com <https://www.namhost.com/>
>
> <https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>
>
> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>
> Namhost Internet Services (Pty) Ltd,
>
> 24 Black Eagle Rd, Hermanus, 7210, RSA
>
> The content of this message is confidential. If you have received it
> by mistake, please inform us by email reply and then delete the
> message. It is forbidden to copy, forward, or in any way reveal the
> contents of this message to anyone without our explicit consent. The
> integrity and security of this email cannot be guaranteed over the
> Internet. Therefore, the sender will not be held liable for any damage
> caused by the message. For our full privacy policy and disclaimers,
> please go to https://www.namhost.com/privacy-policy
>
> Powered by AdSigner
> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>
--
Regards / Groete
<https://www.namhost.com> Granwille Strauss // Senior Systems Admin
*e:* granwille@namhost.com
*m:* +264 81 323 1260 <tel:+264813231260>
*w:* www.namhost.com <https://www.namhost.com/>
<https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
Namhost Internet Services (Pty) Ltd,
24 Black Eagle Rd, Hermanus, 7210, RSA
The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It
is forbidden to copy, forward, or in any way reveal the contents of this
message to anyone without our explicit consent. The integrity and
security of this email cannot be guaranteed over the Internet.
Therefore, the sender will not be held liable for any damage caused by
the message. For our full privacy policy and disclaimers, please go to
https://www.namhost.com/privacy-policy
Powered by AdSigner
<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
Re: Recurring Snaps Failing
Posted by Nicolas Vazquez <Ni...@shapeblue.com>.
Thanks, are the KVM agent and management server running on the same host? Can you increase the log level on the KVM agent logs to DEBUG and retry?
Regards,
Nicolas Vazquez
From: Granwille Strauss <gr...@namhost.com>
Date: Monday, 17 October 2022 at 10:04
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Cc: Nicolas Vazquez <Ni...@shapeblue.com>
Subject: Re: Recurring Snaps Failing
Please find attached TXT
On 10/17/22 14:54, Nicolas Vazquez wrote:
Ok, seems like the command is not reaching the KVM host, can you share the full management server logs for job-796?
Regards,
Nicolas Vazquez
From: Granwille Strauss <gr...@namhost.com>
Date: Monday, 17 October 2022 at 09:32
To: users@cloudstack.apache.org<ma...@cloudstack.apache.org> <us...@cloudstack.apache.org>
Cc: Nicolas Vazquez <Ni...@shapeblue.com>
Subject: Re: Recurring Snaps Failing
Hi
This is all agent threw during the snap process nothing else:
2022-10-17 13:39:12,375 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:a864f9f6) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
2022-10-17 13:40:13,276 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:445216f3) Trying to fetch storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297 from libvirt
2022-10-17 13:40:13,289 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:445216f3) Asking libvirt to refresh storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297
2022-10-17 13:40:13,378 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:445216f3) Trying to fetch storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89 from libvirt
2022-10-17 13:40:13,393 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:445216f3) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
2022-10-17 13:41:13,950 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:87ecfee0) Trying to fetch storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297 from libvirt
2022-10-17 13:41:13,964 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:87ecfee0) Asking libvirt to refresh storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297
2022-10-17 13:41:14,054 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:87ecfee0) Trying to fetch storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89 from libvirt
2022-10-17 13:41:14,067 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:87ecfee0) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
2022-10-17 13:41:57,753 WARN [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-3:null) (logid:d91c35e5) Expected 1 answers while executing SetMonitorServiceCommand but received 3
2022-10-17 13:41:57,938 INFO [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:null) (logid:1549c9a1) Fetching health check result for SEVERIPADDRESS and executing fresh checks: false
On 10/17/22 14:29, Nicolas Vazquez wrote:
Hi Groete,
Can you share the CloudStack agent logs from the KVM host?
Regards,
Nicolas Vazquez
From: Granwille Strauss <gr...@namhost.com.INVALID>
Date: Monday, 17 October 2022 at 08:56
To: users@cloudstack.apache.org<ma...@cloudstack.apache.org> <us...@cloudstack.apache.org>
Subject: Re: Recurring Snaps Failing
Hi
I ran a few more tests and can now finally make sense of the failure:
2022-10-17 13:39:08,556 DEBUG [o.a.c.s.d.d.CloudStackPrimaryDataStoreDriverImpl] (Work-Job-Executor-1:ctx-93cb161b job-806/job-807 ctx-3f9ccf45) (logid:8fd10b20) Failed to take snapshot: 23
2022-10-17 13:39:08,558 DEBUG [o.a.c.s.s.SnapshotServiceImpl] (Work-Job-Executor-1:ctx-93cb161b job-806/job-807 ctx-3f9ccf45) (logid:8fd10b20) create snapshot Test001 failed: com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:1, com.cloud.exception.OperationTimedoutException: Commands 7160441932542378205 to Host 1 timed out after 3600
Seems to be a time out error. But there's no firewalls on the KVM host and NFS server to prevent connections and it worked fine before and randomly just stopped.
So I still have no idea where to solve this as all other necessary settings are in place. Any ideas?
On 10/17/22 08:51, Granwille Strauss wrote:
Hi Guys
I enabled recurring snap shots for my Linux VMs and they all seem to be failing with the following error:
root@hostname ~ $ grep error /var/log/cloudstack/management/management-server.log
2022-10-17 03:01:29,474 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-27:ctx-99d7d702 job-796 ctx-f26a81fe) (logid:f9a61b00) Failed to create snapshot due to an internal error creating snapshot for volume 635fd5b5-bf14-4cf4-979c-be8a649d5189
2022-10-17 03:01:29,476 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-27:ctx-99d7d702 job-796) (logid:f9a61b00) Complete async job-796, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 635fd5b5-bf14-4cf4-979c-be8a649d5189"}
2022-10-17 03:01:29,478 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-26:ctx-c7865967 job-794 ctx-0700afb0) (logid:5541113c) Failed to create snapshot due to an internal error creating snapshot for volume 70ffef79-3aad-43ab-93d1-153fcd34ad15
2022-10-17 03:01:29,480 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-26:ctx-c7865967 job-794) (logid:5541113c) Complete async job-794, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 70ffef79-3aad-43ab-93d1-153fcd34ad15"}
All I am getting out of this is internal error and nothing else. I have checked other logs and I am not quite sure where else to look.
Any assistance in this regard will be appreciated.
--
Regards / Groete
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/>
Granwille Strauss // Senior Systems Admin
e: granwille@namhost.com<ma...@namhost.com>
m: +264 81 323 1260<tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260>
w: www.namhost.com<http://www.namhost.com><http://www.namhost.com><http://www.namhost.com><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/>
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
[https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
Namhost Internet Services (Pty) Ltd,
24 Black Eagle Rd, Hermanus, 7210, RSA
The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go to https://www.namhost.com/privacy-policy
[Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
--
Regards / Groete
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/>
Granwille Strauss // Senior Systems Admin
e: granwille@namhost.com<ma...@namhost.com>
m: +264 81 323 1260<tel:+264813231260><tel:+264813231260><tel:+264813231260><tel:+264813231260>
w: www.namhost.com<http://www.namhost.com><http://www.namhost.com><http://www.namhost.com><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/><https://www.namhost.com/>
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost><https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost><https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
[https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
Namhost Internet Services (Pty) Ltd,
24 Black Eagle Rd, Hermanus, 7210, RSA
The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go to https://www.namhost.com/privacy-policy
[Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
--
Regards / Groete
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/><https://www.namhost.com/>
Granwille Strauss // Senior Systems Admin
e: granwille@namhost.com<ma...@namhost.com>
m: +264 81 323 1260<tel:+264813231260><tel:+264813231260>
w: www.namhost.com<http://www.namhost.com><https://www.namhost.com/><https://www.namhost.com/>
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost><https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost><https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
[https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
Namhost Internet Services (Pty) Ltd,
24 Black Eagle Rd, Hermanus, 7210, RSA
The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go to https://www.namhost.com/privacy-policy
<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
--
Regards / Groete
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
Granwille Strauss // Senior Systems Admin
e: granwille@namhost.com<ma...@namhost.com>
m: +264 81 323 1260<tel:+264813231260>
w: www.namhost.com<https://www.namhost.com/>
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
[https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
Namhost Internet Services (Pty) Ltd,
24 Black Eagle Rd, Hermanus, 7210, RSA
The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go to https://www.namhost.com/privacy-policy
[Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
Re: Recurring Snaps Failing
Posted by Granwille Strauss <gr...@namhost.com.INVALID>.
Please find attached TXT
On 10/17/22 14:54, Nicolas Vazquez wrote:
> Ok, seems like the command is not reaching the KVM host, can you share the full management server logs for job-796?
>
> Regards,
> Nicolas Vazquez
>
>
> From: Granwille Strauss<gr...@namhost.com>
> Date: Monday, 17 October 2022 at 09:32
> To:users@cloudstack.apache.org <us...@cloudstack.apache.org>
> Cc: Nicolas Vazquez<Ni...@shapeblue.com>
> Subject: Re: Recurring Snaps Failing
>
> Hi
>
> This is all agent threw during the snap process nothing else:
> 2022-10-17 13:39:12,375 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:a864f9f6) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
> 2022-10-17 13:40:13,276 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:445216f3) Trying to fetch storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297 from libvirt
> 2022-10-17 13:40:13,289 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:445216f3) Asking libvirt to refresh storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297
> 2022-10-17 13:40:13,378 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:445216f3) Trying to fetch storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89 from libvirt
> 2022-10-17 13:40:13,393 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:445216f3) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
> 2022-10-17 13:41:13,950 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:87ecfee0) Trying to fetch storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297 from libvirt
> 2022-10-17 13:41:13,964 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:87ecfee0) Asking libvirt to refresh storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297
> 2022-10-17 13:41:14,054 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:87ecfee0) Trying to fetch storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89 from libvirt
> 2022-10-17 13:41:14,067 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:87ecfee0) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
> 2022-10-17 13:41:57,753 WARN [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-3:null) (logid:d91c35e5) Expected 1 answers while executing SetMonitorServiceCommand but received 3
> 2022-10-17 13:41:57,938 INFO [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:null) (logid:1549c9a1) Fetching health check result for SEVERIPADDRESS and executing fresh checks: false
>
> On 10/17/22 14:29, Nicolas Vazquez wrote:
>
> Hi Groete,
>
>
>
> Can you share the CloudStack agent logs from the KVM host?
>
>
>
> Regards,
>
> Nicolas Vazquez
>
>
>
>
>
> From: Granwille Strauss<gr...@namhost.com.INVALID>
>
> Date: Monday, 17 October 2022 at 08:56
>
> To:users@cloudstack.apache.org<ma...@cloudstack.apache.org> <us...@cloudstack.apache.org>
>
> Subject: Re: Recurring Snaps Failing
>
>
>
> Hi
>
>
>
> I ran a few more tests and can now finally make sense of the failure:
>
> 2022-10-17 13:39:08,556 DEBUG [o.a.c.s.d.d.CloudStackPrimaryDataStoreDriverImpl] (Work-Job-Executor-1:ctx-93cb161b job-806/job-807 ctx-3f9ccf45) (logid:8fd10b20) Failed to take snapshot: 23
>
> 2022-10-17 13:39:08,558 DEBUG [o.a.c.s.s.SnapshotServiceImpl] (Work-Job-Executor-1:ctx-93cb161b job-806/job-807 ctx-3f9ccf45) (logid:8fd10b20) create snapshot Test001 failed: com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:1, com.cloud.exception.OperationTimedoutException: Commands 7160441932542378205 to Host 1 timed out after 3600
>
>
>
> Seems to be a time out error. But there's no firewalls on the KVM host and NFS server to prevent connections and it worked fine before and randomly just stopped.
>
>
>
> So I still have no idea where to solve this as all other necessary settings are in place. Any ideas?
>
> On 10/17/22 08:51, Granwille Strauss wrote:
>
>
>
> Hi Guys
>
>
>
> I enabled recurring snap shots for my Linux VMs and they all seem to be failing with the following error:
>
> root@hostname ~ $ grep error /var/log/cloudstack/management/management-server.log
>
>
>
>
>
> 2022-10-17 03:01:29,474 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-27:ctx-99d7d702 job-796 ctx-f26a81fe) (logid:f9a61b00) Failed to create snapshot due to an internal error creating snapshot for volume 635fd5b5-bf14-4cf4-979c-be8a649d5189
>
> 2022-10-17 03:01:29,476 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-27:ctx-99d7d702 job-796) (logid:f9a61b00) Complete async job-796, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 635fd5b5-bf14-4cf4-979c-be8a649d5189"}
>
> 2022-10-17 03:01:29,478 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-26:ctx-c7865967 job-794 ctx-0700afb0) (logid:5541113c) Failed to create snapshot due to an internal error creating snapshot for volume 70ffef79-3aad-43ab-93d1-153fcd34ad15
>
> 2022-10-17 03:01:29,480 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-26:ctx-c7865967 job-794) (logid:5541113c) Complete async job-794, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 70ffef79-3aad-43ab-93d1-153fcd34ad15"}
>
>
>
> All I am getting out of this is internal error and nothing else. I have checked other logs and I am not quite sure where else to look.
>
>
>
> Any assistance in this regard will be appreciated.
>
> --
>
>
>
> Regards / Groete
>
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/><https://www.namhost.com/>
>
> Granwille Strauss // Senior Systems Admin
>
>
>
> e:granwille@namhost.com<ma...@namhost.com>
>
> m: +264 81 323 1260<tel:+264813231260><tel:+264813231260>
>
> w:www.namhost.com<http://www.namhost.com><https://www.namhost.com/><https://www.namhost.com/>
>
>
>
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost><https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost><https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>
>
>
> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>
> Namhost Internet Services (Pty) Ltd,
>
>
>
> 24 Black Eagle Rd, Hermanus, 7210, RSA
>
>
>
> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>
>
>
> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>
> --
>
>
>
> Regards / Groete
>
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/><https://www.namhost.com/>
>
> Granwille Strauss // Senior Systems Admin
>
>
>
> e:granwille@namhost.com<ma...@namhost.com>
>
> m: +264 81 323 1260<tel:+264813231260><tel:+264813231260>
>
> w:www.namhost.com<http://www.namhost.com><https://www.namhost.com/><https://www.namhost.com/>
>
>
>
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost><https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost><https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>
>
>
> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>
> Namhost Internet Services (Pty) Ltd,
>
>
>
> 24 Black Eagle Rd, Hermanus, 7210, RSA
>
>
>
> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>
>
>
> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>
>
>
>
>
>
>
>
> --
>
> Regards / Groete
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
> Granwille Strauss // Senior Systems Admin
>
> e:granwille@namhost.com<ma...@namhost.com>
> m: +264 81 323 1260<tel:+264813231260>
> w:www.namhost.com<https://www.namhost.com/>
>
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>
> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
> Namhost Internet Services (Pty) Ltd,
>
> 24 Black Eagle Rd, Hermanus, 7210, RSA
>
> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>
> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>
>
>
>
--
Regards / Groete
<https://www.namhost.com> Granwille Strauss // Senior Systems Admin
*e:* granwille@namhost.com
*m:* +264 81 323 1260 <tel:+264813231260>
*w:* www.namhost.com <https://www.namhost.com/>
<https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
Namhost Internet Services (Pty) Ltd,
24 Black Eagle Rd, Hermanus, 7210, RSA
The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It
is forbidden to copy, forward, or in any way reveal the contents of this
message to anyone without our explicit consent. The integrity and
security of this email cannot be guaranteed over the Internet.
Therefore, the sender will not be held liable for any damage caused by
the message. For our full privacy policy and disclaimers, please go to
https://www.namhost.com/privacy-policy
Powered by AdSigner
<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
Re: Recurring Snaps Failing
Posted by Nicolas Vazquez <Ni...@shapeblue.com>.
Ok, seems like the command is not reaching the KVM host, can you share the full management server logs for job-796?
Regards,
Nicolas Vazquez
From: Granwille Strauss <gr...@namhost.com>
Date: Monday, 17 October 2022 at 09:32
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Cc: Nicolas Vazquez <Ni...@shapeblue.com>
Subject: Re: Recurring Snaps Failing
Hi
This is all agent threw during the snap process nothing else:
2022-10-17 13:39:12,375 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:a864f9f6) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
2022-10-17 13:40:13,276 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:445216f3) Trying to fetch storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297 from libvirt
2022-10-17 13:40:13,289 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:445216f3) Asking libvirt to refresh storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297
2022-10-17 13:40:13,378 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:445216f3) Trying to fetch storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89 from libvirt
2022-10-17 13:40:13,393 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:445216f3) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
2022-10-17 13:41:13,950 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:87ecfee0) Trying to fetch storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297 from libvirt
2022-10-17 13:41:13,964 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:87ecfee0) Asking libvirt to refresh storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297
2022-10-17 13:41:14,054 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:87ecfee0) Trying to fetch storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89 from libvirt
2022-10-17 13:41:14,067 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:87ecfee0) Asking libvirt to refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
2022-10-17 13:41:57,753 WARN [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-3:null) (logid:d91c35e5) Expected 1 answers while executing SetMonitorServiceCommand but received 3
2022-10-17 13:41:57,938 INFO [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:null) (logid:1549c9a1) Fetching health check result for SEVERIPADDRESS and executing fresh checks: false
On 10/17/22 14:29, Nicolas Vazquez wrote:
Hi Groete,
Can you share the CloudStack agent logs from the KVM host?
Regards,
Nicolas Vazquez
From: Granwille Strauss <gr...@namhost.com.INVALID>
Date: Monday, 17 October 2022 at 08:56
To: users@cloudstack.apache.org<ma...@cloudstack.apache.org> <us...@cloudstack.apache.org>
Subject: Re: Recurring Snaps Failing
Hi
I ran a few more tests and can now finally make sense of the failure:
2022-10-17 13:39:08,556 DEBUG [o.a.c.s.d.d.CloudStackPrimaryDataStoreDriverImpl] (Work-Job-Executor-1:ctx-93cb161b job-806/job-807 ctx-3f9ccf45) (logid:8fd10b20) Failed to take snapshot: 23
2022-10-17 13:39:08,558 DEBUG [o.a.c.s.s.SnapshotServiceImpl] (Work-Job-Executor-1:ctx-93cb161b job-806/job-807 ctx-3f9ccf45) (logid:8fd10b20) create snapshot Test001 failed: com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:1, com.cloud.exception.OperationTimedoutException: Commands 7160441932542378205 to Host 1 timed out after 3600
Seems to be a time out error. But there's no firewalls on the KVM host and NFS server to prevent connections and it worked fine before and randomly just stopped.
So I still have no idea where to solve this as all other necessary settings are in place. Any ideas?
On 10/17/22 08:51, Granwille Strauss wrote:
Hi Guys
I enabled recurring snap shots for my Linux VMs and they all seem to be failing with the following error:
root@hostname ~ $ grep error /var/log/cloudstack/management/management-server.log
2022-10-17 03:01:29,474 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-27:ctx-99d7d702 job-796 ctx-f26a81fe) (logid:f9a61b00) Failed to create snapshot due to an internal error creating snapshot for volume 635fd5b5-bf14-4cf4-979c-be8a649d5189
2022-10-17 03:01:29,476 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-27:ctx-99d7d702 job-796) (logid:f9a61b00) Complete async job-796, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 635fd5b5-bf14-4cf4-979c-be8a649d5189"}
2022-10-17 03:01:29,478 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-26:ctx-c7865967 job-794 ctx-0700afb0) (logid:5541113c) Failed to create snapshot due to an internal error creating snapshot for volume 70ffef79-3aad-43ab-93d1-153fcd34ad15
2022-10-17 03:01:29,480 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-26:ctx-c7865967 job-794) (logid:5541113c) Complete async job-794, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 70ffef79-3aad-43ab-93d1-153fcd34ad15"}
All I am getting out of this is internal error and nothing else. I have checked other logs and I am not quite sure where else to look.
Any assistance in this regard will be appreciated.
--
Regards / Groete
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/><https://www.namhost.com/>
Granwille Strauss // Senior Systems Admin
e: granwille@namhost.com<ma...@namhost.com>
m: +264 81 323 1260<tel:+264813231260><tel:+264813231260>
w: www.namhost.com<http://www.namhost.com><https://www.namhost.com/><https://www.namhost.com/>
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost><https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost><https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
[https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
Namhost Internet Services (Pty) Ltd,
24 Black Eagle Rd, Hermanus, 7210, RSA
The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go to https://www.namhost.com/privacy-policy
[Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
--
Regards / Groete
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/><https://www.namhost.com/>
Granwille Strauss // Senior Systems Admin
e: granwille@namhost.com<ma...@namhost.com>
m: +264 81 323 1260<tel:+264813231260><tel:+264813231260>
w: www.namhost.com<http://www.namhost.com><https://www.namhost.com/><https://www.namhost.com/>
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost><https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost><https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/><https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos><https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
[https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner><https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
Namhost Internet Services (Pty) Ltd,
24 Black Eagle Rd, Hermanus, 7210, RSA
The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go to https://www.namhost.com/privacy-policy
[Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818><https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
--
Regards / Groete
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
Granwille Strauss // Senior Systems Admin
e: granwille@namhost.com<ma...@namhost.com>
m: +264 81 323 1260<tel:+264813231260>
w: www.namhost.com<https://www.namhost.com/>
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
[https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
Namhost Internet Services (Pty) Ltd,
24 Black Eagle Rd, Hermanus, 7210, RSA
The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go to https://www.namhost.com/privacy-policy
<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
Re: Recurring Snaps Failing
Posted by Granwille Strauss <gr...@namhost.com.INVALID>.
Hi
This is all agent threw during the snap process nothing else:
> 2022-10-17 13:39:12,375 INFO [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-3:null) (logid:a864f9f6) Asking libvirt to
> refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
> 2022-10-17 13:40:13,276 INFO [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-5:null) (logid:445216f3) Trying to fetch storage
> pool ff9a58d8-fb77-49c8-92d1-7c82fc456297 from libvirt
> 2022-10-17 13:40:13,289 INFO [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-5:null) (logid:445216f3) Asking libvirt to
> refresh storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297
> 2022-10-17 13:40:13,378 INFO [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-3:null) (logid:445216f3) Trying to fetch storage
> pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89 from libvirt
> 2022-10-17 13:40:13,393 INFO [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-3:null) (logid:445216f3) Asking libvirt to
> refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
> 2022-10-17 13:41:13,950 INFO [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-5:null) (logid:87ecfee0) Trying to fetch storage
> pool ff9a58d8-fb77-49c8-92d1-7c82fc456297 from libvirt
> 2022-10-17 13:41:13,964 INFO [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-5:null) (logid:87ecfee0) Asking libvirt to
> refresh storage pool ff9a58d8-fb77-49c8-92d1-7c82fc456297
> 2022-10-17 13:41:14,054 INFO [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-3:null) (logid:87ecfee0) Trying to fetch storage
> pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89 from libvirt
> 2022-10-17 13:41:14,067 INFO [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-3:null) (logid:87ecfee0) Asking libvirt to
> refresh storage pool 81ffca3a-9775-375d-a1c0-9504c0ec3d89
> 2022-10-17 13:41:57,753 WARN
> [resource.virtualnetwork.VirtualRoutingResource]
> (agentRequest-Handler-3:null) (logid:d91c35e5) Expected 1 answers
> while executing SetMonitorServiceCommand but received 3
> 2022-10-17 13:41:57,938 INFO
> [resource.virtualnetwork.VirtualRoutingResource]
> (agentRequest-Handler-1:null) (logid:1549c9a1) Fetching health check
> result for SEVERIPADDRESS and executing fresh checks: false
On 10/17/22 14:29, Nicolas Vazquez wrote:
> Hi Groete,
>
> Can you share the CloudStack agent logs from the KVM host?
>
> Regards,
> Nicolas Vazquez
>
>
> From: Granwille Strauss<gr...@namhost.com.INVALID>
> Date: Monday, 17 October 2022 at 08:56
> To:users@cloudstack.apache.org <us...@cloudstack.apache.org>
> Subject: Re: Recurring Snaps Failing
>
> Hi
>
> I ran a few more tests and can now finally make sense of the failure:
> 2022-10-17 13:39:08,556 DEBUG [o.a.c.s.d.d.CloudStackPrimaryDataStoreDriverImpl] (Work-Job-Executor-1:ctx-93cb161b job-806/job-807 ctx-3f9ccf45) (logid:8fd10b20) Failed to take snapshot: 23
> 2022-10-17 13:39:08,558 DEBUG [o.a.c.s.s.SnapshotServiceImpl] (Work-Job-Executor-1:ctx-93cb161b job-806/job-807 ctx-3f9ccf45) (logid:8fd10b20) create snapshot Test001 failed: com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:1, com.cloud.exception.OperationTimedoutException: Commands 7160441932542378205 to Host 1 timed out after 3600
>
> Seems to be a time out error. But there's no firewalls on the KVM host and NFS server to prevent connections and it worked fine before and randomly just stopped.
>
> So I still have no idea where to solve this as all other necessary settings are in place. Any ideas?
> On 10/17/22 08:51, Granwille Strauss wrote:
>
> Hi Guys
>
> I enabled recurring snap shots for my Linux VMs and they all seem to be failing with the following error:
> root@hostname ~ $ grep error /var/log/cloudstack/management/management-server.log
>
>
> 2022-10-17 03:01:29,474 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-27:ctx-99d7d702 job-796 ctx-f26a81fe) (logid:f9a61b00) Failed to create snapshot due to an internal error creating snapshot for volume 635fd5b5-bf14-4cf4-979c-be8a649d5189
> 2022-10-17 03:01:29,476 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-27:ctx-99d7d702 job-796) (logid:f9a61b00) Complete async job-796, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 635fd5b5-bf14-4cf4-979c-be8a649d5189"}
> 2022-10-17 03:01:29,478 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-26:ctx-c7865967 job-794 ctx-0700afb0) (logid:5541113c) Failed to create snapshot due to an internal error creating snapshot for volume 70ffef79-3aad-43ab-93d1-153fcd34ad15
> 2022-10-17 03:01:29,480 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-26:ctx-c7865967 job-794) (logid:5541113c) Complete async job-794, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 70ffef79-3aad-43ab-93d1-153fcd34ad15"}
>
> All I am getting out of this is internal error and nothing else. I have checked other logs and I am not quite sure where else to look.
>
> Any assistance in this regard will be appreciated.
> --
>
> Regards / Groete
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
> Granwille Strauss // Senior Systems Admin
>
> e:granwille@namhost.com<ma...@namhost.com>
> m: +264 81 323 1260<tel:+264813231260>
> w:www.namhost.com<https://www.namhost.com/>
>
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>
> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
> Namhost Internet Services (Pty) Ltd,
>
> 24 Black Eagle Rd, Hermanus, 7210, RSA
>
> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>
> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
> --
>
> Regards / Groete
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
> Granwille Strauss // Senior Systems Admin
>
> e:granwille@namhost.com<ma...@namhost.com>
> m: +264 81 323 1260<tel:+264813231260>
> w:www.namhost.com<https://www.namhost.com/>
>
> [https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>
> [https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
> Namhost Internet Services (Pty) Ltd,
>
> 24 Black Eagle Rd, Hermanus, 7210, RSA
>
> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>
> [Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>
>
>
>
--
Regards / Groete
<https://www.namhost.com> Granwille Strauss // Senior Systems Admin
*e:* granwille@namhost.com
*m:* +264 81 323 1260 <tel:+264813231260>
*w:* www.namhost.com <https://www.namhost.com/>
<https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
Namhost Internet Services (Pty) Ltd,
24 Black Eagle Rd, Hermanus, 7210, RSA
The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It
is forbidden to copy, forward, or in any way reveal the contents of this
message to anyone without our explicit consent. The integrity and
security of this email cannot be guaranteed over the Internet.
Therefore, the sender will not be held liable for any damage caused by
the message. For our full privacy policy and disclaimers, please go to
https://www.namhost.com/privacy-policy
Powered by AdSigner
<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
Re: Recurring Snaps Failing
Posted by Nicolas Vazquez <Ni...@shapeblue.com>.
Hi Groete,
Can you share the CloudStack agent logs from the KVM host?
Regards,
Nicolas Vazquez
From: Granwille Strauss <gr...@namhost.com.INVALID>
Date: Monday, 17 October 2022 at 08:56
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Re: Recurring Snaps Failing
Hi
I ran a few more tests and can now finally make sense of the failure:
2022-10-17 13:39:08,556 DEBUG [o.a.c.s.d.d.CloudStackPrimaryDataStoreDriverImpl] (Work-Job-Executor-1:ctx-93cb161b job-806/job-807 ctx-3f9ccf45) (logid:8fd10b20) Failed to take snapshot: 23
2022-10-17 13:39:08,558 DEBUG [o.a.c.s.s.SnapshotServiceImpl] (Work-Job-Executor-1:ctx-93cb161b job-806/job-807 ctx-3f9ccf45) (logid:8fd10b20) create snapshot Test001 failed: com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:1, com.cloud.exception.OperationTimedoutException: Commands 7160441932542378205 to Host 1 timed out after 3600
Seems to be a time out error. But there's no firewalls on the KVM host and NFS server to prevent connections and it worked fine before and randomly just stopped.
So I still have no idea where to solve this as all other necessary settings are in place. Any ideas?
On 10/17/22 08:51, Granwille Strauss wrote:
Hi Guys
I enabled recurring snap shots for my Linux VMs and they all seem to be failing with the following error:
root@hostname ~ $ grep error /var/log/cloudstack/management/management-server.log
2022-10-17 03:01:29,474 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-27:ctx-99d7d702 job-796 ctx-f26a81fe) (logid:f9a61b00) Failed to create snapshot due to an internal error creating snapshot for volume 635fd5b5-bf14-4cf4-979c-be8a649d5189
2022-10-17 03:01:29,476 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-27:ctx-99d7d702 job-796) (logid:f9a61b00) Complete async job-796, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 635fd5b5-bf14-4cf4-979c-be8a649d5189"}
2022-10-17 03:01:29,478 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-26:ctx-c7865967 job-794 ctx-0700afb0) (logid:5541113c) Failed to create snapshot due to an internal error creating snapshot for volume 70ffef79-3aad-43ab-93d1-153fcd34ad15
2022-10-17 03:01:29,480 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-26:ctx-c7865967 job-794) (logid:5541113c) Complete async job-794, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 70ffef79-3aad-43ab-93d1-153fcd34ad15"}
All I am getting out of this is internal error and nothing else. I have checked other logs and I am not quite sure where else to look.
Any assistance in this regard will be appreciated.
--
Regards / Groete
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
Granwille Strauss // Senior Systems Admin
e: granwille@namhost.com<ma...@namhost.com>
m: +264 81 323 1260<tel:+264813231260>
w: www.namhost.com<https://www.namhost.com/>
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
[https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
Namhost Internet Services (Pty) Ltd,
24 Black Eagle Rd, Hermanus, 7210, RSA
The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go to https://www.namhost.com/privacy-policy
[Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
--
Regards / Groete
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/logo/621b3fa39fb210001f975298/cd2904ba-304d-4a49-bf33-cbe9ac76d929_248x-.png]<https://www.namhost.com/>
Granwille Strauss // Senior Systems Admin
e: granwille@namhost.com<ma...@namhost.com>
m: +264 81 323 1260<tel:+264813231260>
w: www.namhost.com<https://www.namhost.com/>
[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_01/621b3fa39fb210001f975298/9151954b-b298-41aa-89c8-1d68af075373_48x48.png]<https://www.facebook.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_02/621b3fa39fb210001f975298/85a9dc7c-7bd1-4958-85a9-e6a25baeb028_48x48.png]<https://twitter.com/namhost>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_03/621b3fa39fb210001f975298/c1c5386c-914c-43cf-9d37-5b4aa8e317ab_48x48.png]<https://www.instagram.com/namhostinternetservices/>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_04/621b3fa39fb210001f975298/3aaa7968-130e-48ec-821d-559a332cce47_48x48.png]<https://www.linkedin.com/company/namhos>[https://www.adsigner.com/v1/s/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/social_icon_05/621b3fa39fb210001f975298/3a8c09e6-588f-43a8-acfd-be4423fd3fb6_48x48.png]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
[https://www.adsigner.com/v1/i/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner/940x300]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
Namhost Internet Services (Pty) Ltd,
24 Black Eagle Rd, Hermanus, 7210, RSA
The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go to https://www.namhost.com/privacy-policy
[Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
Re: Recurring Snaps Failing
Posted by Granwille Strauss <gr...@namhost.com.INVALID>.
Hi
I ran a few more tests and can now finally make sense of the failure:
> 2022-10-17 13:39:08,556 DEBUG
> [o.a.c.s.d.d.CloudStackPrimaryDataStoreDriverImpl]
> (Work-Job-Executor-1:ctx-93cb161b job-806/job-807 ctx-3f9ccf45)
> (logid:8fd10b20) Failed to take snapshot: 23
> 2022-10-17 13:39:08,558 DEBUG [o.a.c.s.s.SnapshotServiceImpl]
> (Work-Job-Executor-1:ctx-93cb161b job-806/job-807 ctx-3f9ccf45)
> (logid:8fd10b20) create snapshot Test001 failed:
> com.cloud.utils.exception.CloudRuntimeException: Failed to send
> command, due to Agent:1,
> com.cloud.exception.OperationTimedoutException: Commands
> 7160441932542378205 to Host 1 timed out after 3600
Seems to be a time out error. But there's no firewalls on the KVM host
and NFS server to prevent connections and it worked fine before and
randomly just stopped.
So I still have no idea where to solve this as all other necessary
settings are in place. Any ideas?
On 10/17/22 08:51, Granwille Strauss wrote:
>
> Hi Guys
>
> I enabled recurring snap shots for my Linux VMs and they all seem to
> be failing with the following error:
>
>> root@hostname ~ $ grep error
>> /var/log/cloudstack/management/management-server.log
>
>> 2022-10-17 03:01:29,474 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd]
>> (API-Job-Executor-27:ctx-99d7d702 job-796 ctx-f26a81fe)
>> (logid:f9a61b00) Failed to create snapshot due to an internal error
>> creating snapshot for volume 635fd5b5-bf14-4cf4-979c-be8a649d5189
>> 2022-10-17 03:01:29,476 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-27:ctx-99d7d702 job-796) (logid:f9a61b00) Complete
>> async job-796, jobStatus: FAILED, resultCode: 530, result:
>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed
>> to create snapshot due to an internal error creating snapshot for
>> volume 635fd5b5-bf14-4cf4-979c-be8a649d5189"}
>> 2022-10-17 03:01:29,478 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd]
>> (API-Job-Executor-26:ctx-c7865967 job-794 ctx-0700afb0)
>> (logid:5541113c) Failed to create snapshot due to an internal error
>> creating snapshot for volume 70ffef79-3aad-43ab-93d1-153fcd34ad15
>> 2022-10-17 03:01:29,480 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-26:ctx-c7865967 job-794) (logid:5541113c) Complete
>> async job-794, jobStatus: FAILED, resultCode: 530, result:
>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed
>> to create snapshot due to an internal error creating snapshot for
>> volume 70ffef79-3aad-43ab-93d1-153fcd34ad15"}
>
> All I am getting out of this is internal error and nothing else. I
> have checked other logs and I am not quite sure where else to look.
>
> Any assistance in this regard will be appreciated.
>
> --
> Regards / Groete
>
> <https://www.namhost.com> Granwille Strauss // Senior Systems Admin
>
> *e:* granwille@namhost.com
> *m:* +264 81 323 1260 <tel:+264813231260>
> *w:* www.namhost.com <https://www.namhost.com/>
>
> <https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>
> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
>
> Namhost Internet Services (Pty) Ltd,
> 24 Black Eagle Rd, Hermanus, 7210, RSA
>
>
> The content of this message is confidential. If you have received it
> by mistake, please inform us by email reply and then delete the
> message. It is forbidden to copy, forward, or in any way reveal the
> contents of this message to anyone without our explicit consent. The
> integrity and security of this email cannot be guaranteed over the
> Internet. Therefore, the sender will not be held liable for any damage
> caused by the message. For our full privacy policy and disclaimers,
> please go to https://www.namhost.com/privacy-policy
>
> Powered by AdSigner
> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
--
Regards / Groete
<https://www.namhost.com> Granwille Strauss // Senior Systems Admin
*e:* granwille@namhost.com
*m:* +264 81 323 1260 <tel:+264813231260>
*w:* www.namhost.com <https://www.namhost.com/>
<https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
Namhost Internet Services (Pty) Ltd,
24 Black Eagle Rd, Hermanus, 7210, RSA
The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It
is forbidden to copy, forward, or in any way reveal the contents of this
message to anyone without our explicit consent. The integrity and
security of this email cannot be guaranteed over the Internet.
Therefore, the sender will not be held liable for any damage caused by
the message. For our full privacy policy and disclaimers, please go to
https://www.namhost.com/privacy-policy
Powered by AdSigner
<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>