You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Adam Witwicki <aw...@oakfordis.com> on 2018/11/27 10:11:58 UTC
RE: Help
Hi Guys
All my hosts went down due to a delay in writing to secondary nfs storage,
I have a lot of Virtual Routers in a stuck starting state.
Any tips?
Thanks
Adam
Disclaimer Notice:
This email has been sent by Oakford Technology Limited, while we have checked this e-mail and any attachments for viruses, we can not guarantee that they are virus-free. You must therefore take full responsibility for virus checking.
This message and any attachments are confidential and should only be read by those to whom they are addressed. If you are not the intended recipient, please contact us, delete the message from your computer and destroy any copies. Any distribution or copying without our prior permission is prohibited.
Internet communications are not always secure and therefore Oakford Technology Limited does not accept legal responsibility for this message. The recipient is responsible for verifying its authenticity before acting on the contents. Any views or opinions presented are solely those of the author and do not necessarily represent those of Oakford Technology Limited.
Registered address: Oakford Technology Limited, 10 Prince Maurice Court, Devizes, Wiltshire. SN10 2RT.
Registered in England and Wales No. 5971519
Re: Help
Posted by Andrija Panic <an...@gmail.com>.
OK for VRs
For Instances, check if any of them is running or not running at all (virsh
list, check console of the VMs, etc)- perhaps it's stuck in that "Starting"
state in DB only?
You need to understand if instances are really affected - i.e. are they
really disconnected from the storage, read only FS etc.
Worst case, if instances are really disconnected, you might want to
consider following:
- restart management servers - so any VM starting jobs are killed.
- make sure to all affected VMs are really stopped (kill -9 or similar) on
hypervisors
- revert state of VMs inside DB to "Stopped" - now reality and DB states
are synced.
Now you should be able to cleanly start VM from GUI
VMs that are not stuck, or not affected you will decide if it makes sense
to reboot them or not.
As for remaining VRs - I would consider same story - kill VMs on KVM hosts,
shange state for both VRs to "Stopped" - then restart VPC (old VRs are
deleted, new ones are provisioned) - if this is isolated nework model, then
REstart Network with Clean UP option...
Cheers
On Tue, 27 Nov 2018 at 12:10, Adam Witwicki <aw...@oakfordis.com> wrote:
> The VRs are now up, except for a flappy HA router
>
> Instances are still stuck
>
> -----Original Message-----
> From: Andrija Panic <an...@gmail.com>
> Sent: 27 November 2018 11:05
> To: users <us...@cloudstack.apache.org>
> Subject: Re: Help
>
> ** This mail originated from OUTSIDE the Oakford corporate network. Treat
> hyperlinks and attachments in this email with caution. **
>
> Can you restart VRs, or complete VPC (if VPC) or Restart Network (if
> isolated network, etc)
>
> there is also script for restarting VRs and CPVM and SSVM (i.e. after ACS
> upgrade) if you have many of them.
>
> cloudstack-sysvmadm -d <IP address> -u cloud -p -s
>
>
> hope that helps...
>
> Andrija
>
> On Tue, 27 Nov 2018 at 11:11, Adam Witwicki <aw...@oakfordis.com>
> wrote:
>
> >
> > Hi Guys
> >
> > All my hosts went down due to a delay in writing to secondary nfs
> > storage, I have a lot of Virtual Routers in a stuck starting state.
> >
> > Any tips?
> >
> > Thanks
> >
> > Adam
> >
> >
> >
> > Disclaimer Notice:
> > This email has been sent by Oakford Technology Limited, while we have
> > checked this e-mail and any attachments for viruses, we can not
> > guarantee that they are virus-free. You must therefore take full
> > responsibility for virus checking.
> > This message and any attachments are confidential and should only be
> > read by those to whom they are addressed. If you are not the intended
> > recipient, please contact us, delete the message from your computer
> > and destroy any copies. Any distribution or copying without our prior
> > permission is prohibited.
> > Internet communications are not always secure and therefore Oakford
> > Technology Limited does not accept legal responsibility for this message.
> > The recipient is responsible for verifying its authenticity before
> > acting on the contents. Any views or opinions presented are solely
> > those of the author and do not necessarily represent those of Oakford
> Technology Limited.
> > Registered address: Oakford Technology Limited, 10 Prince Maurice
> > Court, Devizes, Wiltshire. SN10 2RT.
> > Registered in England and Wales No. 5971519
> >
> >
>
> --
>
> Andrija Panić
> Disclaimer Notice:
> This email has been sent by Oakford Technology Limited, while we have
> checked this e-mail and any attachments for viruses, we can not guarantee
> that they are virus-free. You must therefore take full responsibility for
> virus checking.
> This message and any attachments are confidential and should only be read
> by those to whom they are addressed. If you are not the intended recipient,
> please contact us, delete the message from your computer and destroy any
> copies. Any distribution or copying without our prior permission is
> prohibited.
> Internet communications are not always secure and therefore Oakford
> Technology Limited does not accept legal responsibility for this message.
> The recipient is responsible for verifying its authenticity before acting
> on the contents. Any views or opinions presented are solely those of the
> author and do not necessarily represent those of Oakford Technology Limited.
> Registered address: Oakford Technology Limited, 10 Prince Maurice Court,
> Devizes, Wiltshire. SN10 2RT.
> Registered in England and Wales No. 5971519
>
>
--
Andrija Panić
RE: Help
Posted by Adam Witwicki <aw...@oakfordis.com>.
The VRs are now up, except for a flappy HA router
Instances are still stuck
-----Original Message-----
From: Andrija Panic <an...@gmail.com>
Sent: 27 November 2018 11:05
To: users <us...@cloudstack.apache.org>
Subject: Re: Help
** This mail originated from OUTSIDE the Oakford corporate network. Treat hyperlinks and attachments in this email with caution. **
Can you restart VRs, or complete VPC (if VPC) or Restart Network (if isolated network, etc)
there is also script for restarting VRs and CPVM and SSVM (i.e. after ACS
upgrade) if you have many of them.
cloudstack-sysvmadm -d <IP address> -u cloud -p -s
hope that helps...
Andrija
On Tue, 27 Nov 2018 at 11:11, Adam Witwicki <aw...@oakfordis.com> wrote:
>
> Hi Guys
>
> All my hosts went down due to a delay in writing to secondary nfs
> storage, I have a lot of Virtual Routers in a stuck starting state.
>
> Any tips?
>
> Thanks
>
> Adam
>
>
>
> Disclaimer Notice:
> This email has been sent by Oakford Technology Limited, while we have
> checked this e-mail and any attachments for viruses, we can not
> guarantee that they are virus-free. You must therefore take full
> responsibility for virus checking.
> This message and any attachments are confidential and should only be
> read by those to whom they are addressed. If you are not the intended
> recipient, please contact us, delete the message from your computer
> and destroy any copies. Any distribution or copying without our prior
> permission is prohibited.
> Internet communications are not always secure and therefore Oakford
> Technology Limited does not accept legal responsibility for this message.
> The recipient is responsible for verifying its authenticity before
> acting on the contents. Any views or opinions presented are solely
> those of the author and do not necessarily represent those of Oakford Technology Limited.
> Registered address: Oakford Technology Limited, 10 Prince Maurice
> Court, Devizes, Wiltshire. SN10 2RT.
> Registered in England and Wales No. 5971519
>
>
--
Andrija Panić
Disclaimer Notice:
This email has been sent by Oakford Technology Limited, while we have checked this e-mail and any attachments for viruses, we can not guarantee that they are virus-free. You must therefore take full responsibility for virus checking.
This message and any attachments are confidential and should only be read by those to whom they are addressed. If you are not the intended recipient, please contact us, delete the message from your computer and destroy any copies. Any distribution or copying without our prior permission is prohibited.
Internet communications are not always secure and therefore Oakford Technology Limited does not accept legal responsibility for this message. The recipient is responsible for verifying its authenticity before acting on the contents. Any views or opinions presented are solely those of the author and do not necessarily represent those of Oakford Technology Limited.
Registered address: Oakford Technology Limited, 10 Prince Maurice Court, Devizes, Wiltshire. SN10 2RT.
Registered in England and Wales No. 5971519
Re: Help
Posted by Andrija Panic <an...@gmail.com>.
Can you restart VRs, or complete VPC (if VPC) or Restart Network (if
isolated network, etc)
there is also script for restarting VRs and CPVM and SSVM (i.e. after ACS
upgrade) if you have many of them.
cloudstack-sysvmadm -d <IP address> -u cloud -p -s
hope that helps...
Andrija
On Tue, 27 Nov 2018 at 11:11, Adam Witwicki <aw...@oakfordis.com> wrote:
>
> Hi Guys
>
> All my hosts went down due to a delay in writing to secondary nfs storage,
> I have a lot of Virtual Routers in a stuck starting state.
>
> Any tips?
>
> Thanks
>
> Adam
>
>
>
> Disclaimer Notice:
> This email has been sent by Oakford Technology Limited, while we have
> checked this e-mail and any attachments for viruses, we can not guarantee
> that they are virus-free. You must therefore take full responsibility for
> virus checking.
> This message and any attachments are confidential and should only be read
> by those to whom they are addressed. If you are not the intended recipient,
> please contact us, delete the message from your computer and destroy any
> copies. Any distribution or copying without our prior permission is
> prohibited.
> Internet communications are not always secure and therefore Oakford
> Technology Limited does not accept legal responsibility for this message.
> The recipient is responsible for verifying its authenticity before acting
> on the contents. Any views or opinions presented are solely those of the
> author and do not necessarily represent those of Oakford Technology Limited.
> Registered address: Oakford Technology Limited, 10 Prince Maurice Court,
> Devizes, Wiltshire. SN10 2RT.
> Registered in England and Wales No. 5971519
>
>
--
Andrija Panić
RE: Help
Posted by Adam Witwicki <aw...@oakfordis.com>.
Andrija,
You are correct, I was assuming I only used nfs on secondary storage, I also have primary nfs storage for the VR system offering, Doh
I feel that modifying the script to shout a lot and not reboot will help us in the first instance
Thanks :)
Adam
-----Original Message-----
From: Andrija Panic <an...@gmail.com>
Sent: 27 November 2018 12:07
To: users <us...@cloudstack.apache.org>
Subject: Re: Help
** This mail originated from OUTSIDE the Oakford corporate network. Treat hyperlinks and attachments in this email with caution. **
/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh
comment the 6th from the bottom line: echo b > /proc/sysrq-trigger
That stops reboot process - been there, done that for years now :)
But then investigate what is going on - since it's not "normal" to trigger this script (reboot actually) in healthy environment.
FYI, there is no such heartbeat checks for CEPH, afaik.
It's strange, that this also affects Secondary Storage NFS - I had idea it's only made for Primary Storage - actually I'm almost sure it's NOT checking Secondary Storage, since on PRimary NFS there is automatically created folder KVMHA, and inside it a file for each KVM node which should be updated (modification time) revery minute if KVM can access the NFS...
Cheers
On Tue, 27 Nov 2018 at 12:56, Adam Witwicki <aw...@oakfordis.com> wrote:
> There seems to have been an issue with connectivity on a host that was
> causing the VRs to silently fail.
>
> How can I stop this script rebooting when it detects an issue with NFS?
> Would setting up another NFS pool work?
>
> Nov 27 08:23:41 OIS-MH-P1-C1-H1-C heartbeat: kvmheartbeat.sh rebooted
> system because it was unable to write the heartbeat to the storage.
>
>
> Thanks
>
> Adam
>
> -----Original Message-----
> From: Adam Witwicki <aw...@oakfordis.com>
> Sent: 27 November 2018 11:09
> To: users@cloudstack.apache.org
> Subject: RE: Help
>
> ** This mail originated from OUTSIDE the Oakford corporate network.
> Treat hyperlinks and attachments in this email with caution. **
>
> Hi Swen
>
> I use ceph for primary storage
> And NFS for secondary storage
>
> Using KVM hosts I got this error
> Nov 27 08:23:41 OIS-MH-P1-C1-H1-C heartbeat: kvmheartbeat.sh rebooted
> system because it was unable to write the heartbeat to the storage.
>
> I fixed the issue with NFS
> And now a lot of instances are stuck in starting state, after the
> hosts went into a reboot flap
>
> Thanks
>
> Adam
>
> -----Original Message-----
> From: Swen - swen.io <me...@swen.io>
> Sent: 27 November 2018 11:05
> To: users@cloudstack.apache.org
> Subject: AW: Help
>
> ** This mail originated from OUTSIDE the Oakford corporate network.
> Treat hyperlinks and attachments in this email with caution. **
>
> Hi Adam,
>
> can you please explain a little bit more why your VR are stuck because
> of your problem with secondary storage? Are you using the same storage
> as primary storage for your VRs?
> Is the problem already fixed with your secondary storage? Can you
> restart a network?
>
> Cu Swen
>
> -----Ursprüngliche Nachricht-----
> Von: Adam Witwicki <aw...@oakfordis.com>
> Gesendet: Dienstag, 27. November 2018 11:12
> An: users@cloudstack.apache.org
> Betreff: RE: Help
>
>
> Hi Guys
>
> All my hosts went down due to a delay in writing to secondary nfs
> storage, I have a lot of Virtual Routers in a stuck starting state.
>
> Any tips?
>
> Thanks
>
> Adam
>
>
>
> Disclaimer Notice:
> This email has been sent by Oakford Technology Limited, while we have
> checked this e-mail and any attachments for viruses, we can not
> guarantee that they are virus-free. You must therefore take full
> responsibility for virus checking.
> This message and any attachments are confidential and should only be
> read by those to whom they are addressed. If you are not the intended
> recipient, please contact us, delete the message from your computer
> and destroy any copies. Any distribution or copying without our prior
> permission is prohibited.
> Internet communications are not always secure and therefore Oakford
> Technology Limited does not accept legal responsibility for this message.
> The recipient is responsible for verifying its authenticity before
> acting on the contents. Any views or opinions presented are solely
> those of the author and do not necessarily represent those of Oakford Technology Limited.
> Registered address: Oakford Technology Limited, 10 Prince Maurice
> Court, Devizes, Wiltshire. SN10 2RT.
> Registered in England and Wales No. 5971519
>
>
>
> Disclaimer Notice:
> This email has been sent by Oakford Technology Limited, while we have
> checked this e-mail and any attachments for viruses, we can not
> guarantee that they are virus-free. You must therefore take full
> responsibility for virus checking.
> This message and any attachments are confidential and should only be
> read by those to whom they are addressed. If you are not the intended
> recipient, please contact us, delete the message from your computer
> and destroy any copies. Any distribution or copying without our prior
> permission is prohibited.
> Internet communications are not always secure and therefore Oakford
> Technology Limited does not accept legal responsibility for this message.
> The recipient is responsible for verifying its authenticity before
> acting on the contents. Any views or opinions presented are solely
> those of the author and do not necessarily represent those of Oakford Technology Limited.
> Registered address: Oakford Technology Limited, 10 Prince Maurice
> Court, Devizes, Wiltshire. SN10 2RT.
> Registered in England and Wales No. 5971519
>
> Disclaimer Notice:
> This email has been sent by Oakford Technology Limited, while we have
> checked this e-mail and any attachments for viruses, we can not
> guarantee that they are virus-free. You must therefore take full
> responsibility for virus checking.
> This message and any attachments are confidential and should only be
> read by those to whom they are addressed. If you are not the intended
> recipient, please contact us, delete the message from your computer
> and destroy any copies. Any distribution or copying without our prior
> permission is prohibited.
> Internet communications are not always secure and therefore Oakford
> Technology Limited does not accept legal responsibility for this message.
> The recipient is responsible for verifying its authenticity before
> acting on the contents. Any views or opinions presented are solely
> those of the author and do not necessarily represent those of Oakford Technology Limited.
> Registered address: Oakford Technology Limited, 10 Prince Maurice
> Court, Devizes, Wiltshire. SN10 2RT.
> Registered in England and Wales No. 5971519
>
>
--
Andrija Panić
Disclaimer Notice:
This email has been sent by Oakford Technology Limited, while we have checked this e-mail and any attachments for viruses, we can not guarantee that they are virus-free. You must therefore take full responsibility for virus checking.
This message and any attachments are confidential and should only be read by those to whom they are addressed. If you are not the intended recipient, please contact us, delete the message from your computer and destroy any copies. Any distribution or copying without our prior permission is prohibited.
Internet communications are not always secure and therefore Oakford Technology Limited does not accept legal responsibility for this message. The recipient is responsible for verifying its authenticity before acting on the contents. Any views or opinions presented are solely those of the author and do not necessarily represent those of Oakford Technology Limited.
Registered address: Oakford Technology Limited, 10 Prince Maurice Court, Devizes, Wiltshire. SN10 2RT.
Registered in England and Wales No. 5971519
Re: Help
Posted by Andrija Panic <an...@gmail.com>.
/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh
comment the 6th from the bottom line: echo b > /proc/sysrq-trigger
That stops reboot process - been there, done that for years now :)
But then investigate what is going on - since it's not "normal" to trigger
this script (reboot actually) in healthy environment.
FYI, there is no such heartbeat checks for CEPH, afaik.
It's strange, that this also affects Secondary Storage NFS - I had idea
it's only made for Primary Storage - actually I'm almost sure it's NOT
checking Secondary Storage, since on PRimary NFS there is automatically
created folder KVMHA, and inside it a file for each KVM node which should
be updated (modification time) revery minute if KVM can access the NFS...
Cheers
On Tue, 27 Nov 2018 at 12:56, Adam Witwicki <aw...@oakfordis.com> wrote:
> There seems to have been an issue with connectivity on a host that was
> causing the VRs to silently fail.
>
> How can I stop this script rebooting when it detects an issue with NFS?
> Would setting up another NFS pool work?
>
> Nov 27 08:23:41 OIS-MH-P1-C1-H1-C heartbeat: kvmheartbeat.sh rebooted
> system because it was unable to write the heartbeat to the storage.
>
>
> Thanks
>
> Adam
>
> -----Original Message-----
> From: Adam Witwicki <aw...@oakfordis.com>
> Sent: 27 November 2018 11:09
> To: users@cloudstack.apache.org
> Subject: RE: Help
>
> ** This mail originated from OUTSIDE the Oakford corporate network. Treat
> hyperlinks and attachments in this email with caution. **
>
> Hi Swen
>
> I use ceph for primary storage
> And NFS for secondary storage
>
> Using KVM hosts I got this error
> Nov 27 08:23:41 OIS-MH-P1-C1-H1-C heartbeat: kvmheartbeat.sh rebooted
> system because it was unable to write the heartbeat to the storage.
>
> I fixed the issue with NFS
> And now a lot of instances are stuck in starting state, after the hosts
> went into a reboot flap
>
> Thanks
>
> Adam
>
> -----Original Message-----
> From: Swen - swen.io <me...@swen.io>
> Sent: 27 November 2018 11:05
> To: users@cloudstack.apache.org
> Subject: AW: Help
>
> ** This mail originated from OUTSIDE the Oakford corporate network. Treat
> hyperlinks and attachments in this email with caution. **
>
> Hi Adam,
>
> can you please explain a little bit more why your VR are stuck because of
> your problem with secondary storage? Are you using the same storage as
> primary storage for your VRs?
> Is the problem already fixed with your secondary storage? Can you restart
> a network?
>
> Cu Swen
>
> -----Ursprüngliche Nachricht-----
> Von: Adam Witwicki <aw...@oakfordis.com>
> Gesendet: Dienstag, 27. November 2018 11:12
> An: users@cloudstack.apache.org
> Betreff: RE: Help
>
>
> Hi Guys
>
> All my hosts went down due to a delay in writing to secondary nfs storage,
> I have a lot of Virtual Routers in a stuck starting state.
>
> Any tips?
>
> Thanks
>
> Adam
>
>
>
> Disclaimer Notice:
> This email has been sent by Oakford Technology Limited, while we have
> checked this e-mail and any attachments for viruses, we can not guarantee
> that they are virus-free. You must therefore take full responsibility for
> virus checking.
> This message and any attachments are confidential and should only be read
> by those to whom they are addressed. If you are not the intended recipient,
> please contact us, delete the message from your computer and destroy any
> copies. Any distribution or copying without our prior permission is
> prohibited.
> Internet communications are not always secure and therefore Oakford
> Technology Limited does not accept legal responsibility for this message.
> The recipient is responsible for verifying its authenticity before acting
> on the contents. Any views or opinions presented are solely those of the
> author and do not necessarily represent those of Oakford Technology Limited.
> Registered address: Oakford Technology Limited, 10 Prince Maurice Court,
> Devizes, Wiltshire. SN10 2RT.
> Registered in England and Wales No. 5971519
>
>
>
> Disclaimer Notice:
> This email has been sent by Oakford Technology Limited, while we have
> checked this e-mail and any attachments for viruses, we can not guarantee
> that they are virus-free. You must therefore take full responsibility for
> virus checking.
> This message and any attachments are confidential and should only be read
> by those to whom they are addressed. If you are not the intended recipient,
> please contact us, delete the message from your computer and destroy any
> copies. Any distribution or copying without our prior permission is
> prohibited.
> Internet communications are not always secure and therefore Oakford
> Technology Limited does not accept legal responsibility for this message.
> The recipient is responsible for verifying its authenticity before acting
> on the contents. Any views or opinions presented are solely those of the
> author and do not necessarily represent those of Oakford Technology Limited.
> Registered address: Oakford Technology Limited, 10 Prince Maurice Court,
> Devizes, Wiltshire. SN10 2RT.
> Registered in England and Wales No. 5971519
>
> Disclaimer Notice:
> This email has been sent by Oakford Technology Limited, while we have
> checked this e-mail and any attachments for viruses, we can not guarantee
> that they are virus-free. You must therefore take full responsibility for
> virus checking.
> This message and any attachments are confidential and should only be read
> by those to whom they are addressed. If you are not the intended recipient,
> please contact us, delete the message from your computer and destroy any
> copies. Any distribution or copying without our prior permission is
> prohibited.
> Internet communications are not always secure and therefore Oakford
> Technology Limited does not accept legal responsibility for this message.
> The recipient is responsible for verifying its authenticity before acting
> on the contents. Any views or opinions presented are solely those of the
> author and do not necessarily represent those of Oakford Technology Limited.
> Registered address: Oakford Technology Limited, 10 Prince Maurice Court,
> Devizes, Wiltshire. SN10 2RT.
> Registered in England and Wales No. 5971519
>
>
--
Andrija Panić
RE: Help
Posted by Adam Witwicki <aw...@oakfordis.com>.
There seems to have been an issue with connectivity on a host that was causing the VRs to silently fail.
How can I stop this script rebooting when it detects an issue with NFS? Would setting up another NFS pool work?
Nov 27 08:23:41 OIS-MH-P1-C1-H1-C heartbeat: kvmheartbeat.sh rebooted system because it was unable to write the heartbeat to the storage.
Thanks
Adam
-----Original Message-----
From: Adam Witwicki <aw...@oakfordis.com>
Sent: 27 November 2018 11:09
To: users@cloudstack.apache.org
Subject: RE: Help
** This mail originated from OUTSIDE the Oakford corporate network. Treat hyperlinks and attachments in this email with caution. **
Hi Swen
I use ceph for primary storage
And NFS for secondary storage
Using KVM hosts I got this error
Nov 27 08:23:41 OIS-MH-P1-C1-H1-C heartbeat: kvmheartbeat.sh rebooted system because it was unable to write the heartbeat to the storage.
I fixed the issue with NFS
And now a lot of instances are stuck in starting state, after the hosts went into a reboot flap
Thanks
Adam
-----Original Message-----
From: Swen - swen.io <me...@swen.io>
Sent: 27 November 2018 11:05
To: users@cloudstack.apache.org
Subject: AW: Help
** This mail originated from OUTSIDE the Oakford corporate network. Treat hyperlinks and attachments in this email with caution. **
Hi Adam,
can you please explain a little bit more why your VR are stuck because of your problem with secondary storage? Are you using the same storage as primary storage for your VRs?
Is the problem already fixed with your secondary storage? Can you restart a network?
Cu Swen
-----Ursprüngliche Nachricht-----
Von: Adam Witwicki <aw...@oakfordis.com>
Gesendet: Dienstag, 27. November 2018 11:12
An: users@cloudstack.apache.org
Betreff: RE: Help
Hi Guys
All my hosts went down due to a delay in writing to secondary nfs storage, I have a lot of Virtual Routers in a stuck starting state.
Any tips?
Thanks
Adam
Disclaimer Notice:
This email has been sent by Oakford Technology Limited, while we have checked this e-mail and any attachments for viruses, we can not guarantee that they are virus-free. You must therefore take full responsibility for virus checking.
This message and any attachments are confidential and should only be read by those to whom they are addressed. If you are not the intended recipient, please contact us, delete the message from your computer and destroy any copies. Any distribution or copying without our prior permission is prohibited.
Internet communications are not always secure and therefore Oakford Technology Limited does not accept legal responsibility for this message.
The recipient is responsible for verifying its authenticity before acting on the contents. Any views or opinions presented are solely those of the author and do not necessarily represent those of Oakford Technology Limited.
Registered address: Oakford Technology Limited, 10 Prince Maurice Court, Devizes, Wiltshire. SN10 2RT.
Registered in England and Wales No. 5971519
Disclaimer Notice:
This email has been sent by Oakford Technology Limited, while we have checked this e-mail and any attachments for viruses, we can not guarantee that they are virus-free. You must therefore take full responsibility for virus checking.
This message and any attachments are confidential and should only be read by those to whom they are addressed. If you are not the intended recipient, please contact us, delete the message from your computer and destroy any copies. Any distribution or copying without our prior permission is prohibited.
Internet communications are not always secure and therefore Oakford Technology Limited does not accept legal responsibility for this message. The recipient is responsible for verifying its authenticity before acting on the contents. Any views or opinions presented are solely those of the author and do not necessarily represent those of Oakford Technology Limited.
Registered address: Oakford Technology Limited, 10 Prince Maurice Court, Devizes, Wiltshire. SN10 2RT.
Registered in England and Wales No. 5971519
Disclaimer Notice:
This email has been sent by Oakford Technology Limited, while we have checked this e-mail and any attachments for viruses, we can not guarantee that they are virus-free. You must therefore take full responsibility for virus checking.
This message and any attachments are confidential and should only be read by those to whom they are addressed. If you are not the intended recipient, please contact us, delete the message from your computer and destroy any copies. Any distribution or copying without our prior permission is prohibited.
Internet communications are not always secure and therefore Oakford Technology Limited does not accept legal responsibility for this message. The recipient is responsible for verifying its authenticity before acting on the contents. Any views or opinions presented are solely those of the author and do not necessarily represent those of Oakford Technology Limited.
Registered address: Oakford Technology Limited, 10 Prince Maurice Court, Devizes, Wiltshire. SN10 2RT.
Registered in England and Wales No. 5971519
RE: Help
Posted by Adam Witwicki <aw...@oakfordis.com>.
Hi Swen
I use ceph for primary storage
And NFS for secondary storage
Using KVM hosts I got this error
Nov 27 08:23:41 OIS-MH-P1-C1-H1-C heartbeat: kvmheartbeat.sh rebooted system because it was unable to write the heartbeat to the storage.
I fixed the issue with NFS
And now a lot of instances are stuck in starting state, after the hosts went into a reboot flap
Thanks
Adam
-----Original Message-----
From: Swen - swen.io <me...@swen.io>
Sent: 27 November 2018 11:05
To: users@cloudstack.apache.org
Subject: AW: Help
** This mail originated from OUTSIDE the Oakford corporate network. Treat hyperlinks and attachments in this email with caution. **
Hi Adam,
can you please explain a little bit more why your VR are stuck because of your problem with secondary storage? Are you using the same storage as primary storage for your VRs?
Is the problem already fixed with your secondary storage? Can you restart a network?
Cu Swen
-----Ursprüngliche Nachricht-----
Von: Adam Witwicki <aw...@oakfordis.com>
Gesendet: Dienstag, 27. November 2018 11:12
An: users@cloudstack.apache.org
Betreff: RE: Help
Hi Guys
All my hosts went down due to a delay in writing to secondary nfs storage, I have a lot of Virtual Routers in a stuck starting state.
Any tips?
Thanks
Adam
Disclaimer Notice:
This email has been sent by Oakford Technology Limited, while we have checked this e-mail and any attachments for viruses, we can not guarantee that they are virus-free. You must therefore take full responsibility for virus checking.
This message and any attachments are confidential and should only be read by those to whom they are addressed. If you are not the intended recipient, please contact us, delete the message from your computer and destroy any copies. Any distribution or copying without our prior permission is prohibited.
Internet communications are not always secure and therefore Oakford Technology Limited does not accept legal responsibility for this message.
The recipient is responsible for verifying its authenticity before acting on the contents. Any views or opinions presented are solely those of the author and do not necessarily represent those of Oakford Technology Limited.
Registered address: Oakford Technology Limited, 10 Prince Maurice Court, Devizes, Wiltshire. SN10 2RT.
Registered in England and Wales No. 5971519
Disclaimer Notice:
This email has been sent by Oakford Technology Limited, while we have checked this e-mail and any attachments for viruses, we can not guarantee that they are virus-free. You must therefore take full responsibility for virus checking.
This message and any attachments are confidential and should only be read by those to whom they are addressed. If you are not the intended recipient, please contact us, delete the message from your computer and destroy any copies. Any distribution or copying without our prior permission is prohibited.
Internet communications are not always secure and therefore Oakford Technology Limited does not accept legal responsibility for this message. The recipient is responsible for verifying its authenticity before acting on the contents. Any views or opinions presented are solely those of the author and do not necessarily represent those of Oakford Technology Limited.
Registered address: Oakford Technology Limited, 10 Prince Maurice Court, Devizes, Wiltshire. SN10 2RT.
Registered in England and Wales No. 5971519
AW: Help
Posted by "Swen - swen.io" <me...@swen.io>.
Hi Adam,
can you please explain a little bit more why your VR are stuck because of
your problem with secondary storage? Are you using the same storage as
primary storage for your VRs?
Is the problem already fixed with your secondary storage? Can you restart a
network?
Cu Swen
-----Ursprüngliche Nachricht-----
Von: Adam Witwicki <aw...@oakfordis.com>
Gesendet: Dienstag, 27. November 2018 11:12
An: users@cloudstack.apache.org
Betreff: RE: Help
Hi Guys
All my hosts went down due to a delay in writing to secondary nfs storage, I
have a lot of Virtual Routers in a stuck starting state.
Any tips?
Thanks
Adam
Disclaimer Notice:
This email has been sent by Oakford Technology Limited, while we have
checked this e-mail and any attachments for viruses, we can not guarantee
that they are virus-free. You must therefore take full responsibility for
virus checking.
This message and any attachments are confidential and should only be read by
those to whom they are addressed. If you are not the intended recipient,
please contact us, delete the message from your computer and destroy any
copies. Any distribution or copying without our prior permission is
prohibited.
Internet communications are not always secure and therefore Oakford
Technology Limited does not accept legal responsibility for this message.
The recipient is responsible for verifying its authenticity before acting on
the contents. Any views or opinions presented are solely those of the author
and do not necessarily represent those of Oakford Technology Limited.
Registered address: Oakford Technology Limited, 10 Prince Maurice Court,
Devizes, Wiltshire. SN10 2RT.
Registered in England and Wales No. 5971519