You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@cloudstack.apache.org by Sven Vogel <S....@ewerk.com> on 2020/06/01 11:29:34 UTC

Managed Storage and HA

Hi Community,

I try to encounter how HA works. Our goal is it to make it usable with managed storage like (Netapp Solidfire / maybe it works with CEPH too) so if its possible.

This is a good guide and for some times we fixed and added the missing keys.
https://cwiki.apache.org/confluence/display/CLOUDSTACK/High+Availability+Developer%27s+Guide<https://cwiki.apache.org/confluence/display/CLOUDSTACK/High+Availability+Developer's+Guide>
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Host+HA

In the database I found out there are three different types of HA.

If you select in the configuration table "SELECT * FROM `configuration` WHERE `component` LIKE '%ha%' LIMIT 0,1000;“ you will get three types of components.

1. HAManager
2. HighAvailbilityManager
3. KVMHAConfig

"HAManager and HighAvailbilityManager" are the base which was extended from Rohit with „KVMHAConfig“ - KVM with stonith fencing.

I understand all things work together but maybe I need to understand the process a little bit better.

------------------------------------------------------------------------------------
To clarify I write down what I think to each of them. This is what I understand but please correct me or help me to understand it a little bit better.

—>I found out that if we use managed storage a restart of virtual machines only works on the same host. This is what I understand the lack of the missing heartbeat file on the shared storage because we don’t have shared storage like NFS.

—
"If the network ping investigation returns that it cannot detect the status of the host, CloudStack HA then relies on the hypervisor specific investigation. For VMware, there is no such investigation as the hypervisor host handles its own HA. For XenServer and KVM, CloudStack HA deploys a monitoring script that writes the current timestamp on to a heartbeat file on shared storage."
—

And

—
For the initial release, only KVM with NFS storage will be supported. However, the storage check component will be implemented in a modular fashion allowing for checks using other storage platforms(e.g. Ceph) in the future.
—
------------------------------------------------------------------------------------

We would implement a plugin or extend this for managed storage but at the moment I need to understand where this should happen. Since managed storage uses different volumes for each VM we should its not easy to make an storage heartbeat like NFS. the lack of one missing volume don’t means the hole storage has an problem so I think its not easy to encounter from one volumes to a complete storage.

We don’t use KVMHAConfig a the moment and encounter that if a Host goes down (offline) the virtual machines will not be restarted on another host. They will only restarted on the host if the host comes back.(online). We don’t want a hard fencing of the hosts but we want a correct determination whether the host is still alive. Fencing would maybe in our case a little bit hard because we don’t have an hard data corruption on entire storage.

Some questions.
1. let's assume correctly that the HA don’t work without an shared storage and network ping? Is this the cause why our virtual machines will not restarted on another host? Is this correct or do we have an config problem?
2. Where could the plugin be implemented? Is there a preferred place?
3. If point 1. Is correctly I thought the idea would be to add an global flag to use the secondary storage (NFS) as heartbeat to find out if there is any host inactive?

Thanks and Cheers

Sven

Sven Vogel
Lead Cloud Solution Architect

EWERK DIGITAL GmbH
Brühl 24, D-04109 Leipzig
P +49 341 42649 - 99
F +49 341 42649 - 98
S.Vogel@ewerk.com
www.ewerk.com

Geschäftsführer:
Dr. Erik Wende, Hendrik Schubert, Tassilo Möschke
Registergericht: Leipzig HRB 9065

Support:
+49 341 42649 555

Zertifiziert nach:
ISO/IEC 27001:2013
DIN EN ISO 9001:2015
DIN ISO/IEC 20000-1:2011

ISAE 3402 Typ II Assessed

EWERK-Blog<https://blog.ewerk.com/> | LinkedIn<https://www.linkedin.com/company/ewerk-group> | Xing<https://www.xing.com/company/ewerk> | Twitter<https://twitter.com/EWERK_Group> | Facebook<https://de-de.facebook.com/EWERK.IT/>

Auskünfte und Angebote per Mail sind freibleibend und unverbindlich.

Disclaimer Privacy:
Der Inhalt dieser E-Mail (einschließlich etwaiger beigefügter Dateien) ist vertraulich und nur für den Empfänger bestimmt. Sollten Sie nicht der bestimmungsgemäße Empfänger sein, ist Ihnen jegliche Offenlegung, Vervielfältigung, Weitergabe oder Nutzung des Inhalts untersagt. Bitte informieren Sie in diesem Fall unverzüglich den Absender und löschen Sie die E-Mail (einschließlich etwaiger beigefügter Dateien) von Ihrem System. Vielen Dank.

The contents of this e-mail (including any attachments) are confidential and may be legally privileged. If you are not the intended recipient of this e-mail, any disclosure, copying, distribution or use of its contents is strictly prohibited, and you should please notify the sender immediately and then delete it (including any attachments) from your system. Thank you.

Re: Managed Storage and HA

Posted by "Tutkowski, Mike" <Mi...@netapp.com>.

I am thinking you would make the ListVolumeStats API call to the SF cluster from a management server because we know that management servers should have access to the management network of the SF cluster. I know some KVM hosts might have access to the management network of the SF cluster that supports their volumes, but they don't have to: KVM hosts (and hypervisor hosts in general) only need access to the storage network of the SF cluster that supports their volumes. If the agent happened to be running on a KVM host that only has access to the storage network, then this agent can't make the ListVolumeStats API call. As such, I think the ListVolumeStats API call should be executed from a management server.

You are correct that the volumes_details tables contains the ID of a SolidFire volume.

I don't know all of the ways a volume can transition from the Ready state to some other state. I would not rely on the volume's state in the DB to determine if a VM is having trouble accessing a volume.

On 6/8/20, 2:47 PM, "Sven Vogel" <S....@ewerk.com> wrote:

NetApp Security WARNING: This is an external email. Do not click links or open attachments unless you recognize the sender and know the content is safe.

Hi Paul, Hi Mike, Hi Syed,

@Paul
> Doesn’t the same problem exist with that? If the agent dies, then the fake volume will stop being updated but the VMs would still be running.

Do you mean we don’t know if a machine is running and get an false positive? I think about if we had only one fake volume and if they are not there so we would get an fencing of the host and kill all machines. Maybe the other volumes are accessible.

I see different challenges.

1. SF uses normally a storage network so with that if the management network is down don’t means the volumes are not accessible.
2. A Fake volume can produce false positives maybe this one is due too any reason will not accessible but the other ones can be achieved

@Mike

> Since you know which (SolidFire) volumes are connected to a particular KVM host, I would think you could just use the ListVolumeStats API (described below) from a management server to determine if there is disk activity to any of those volumes.

You mean the work would be done not from the Agent but from the management. You mean we should periodically check the "ListVolumeStats“ on the Solidfire from the Management Server and know if they are active.

What I see is that in the „volume_details“ table are all SF volumes are listed. Right? They are connected with the „volumes“ table and there I will finde the „instance_id“. Now there are two ways. Find about the „vm_instance“ table the if the instance is „running“, I think this is important and „host_id“

I don’t know if the „volumes“ table „ready“ state is useful. This will give us a „ready“ state but does not mean its active. I hope I am correct.

@Paul
If this is thinkable where is a best way to implement it that it works. I mean what I need to enable that i know HA is working but lacking on the storage heartbeat.

1. How I need to configure HA that it will work?
2. I read the documentation from Rohit and found that he wrote "For the initial release, only KVM with NFS storage will be supported. However, the storage check component will be implemented in a modular fashion allowing for checks using other storage platforms(e.g. Ceph) in the future.“ Where is a good way In the code to start so won’t implement it wrong and lets more focused on finalizing? Maybe you can clarify it a little bit.

Cheers Sven

Sven Vogel
Lead Cloud Solution Architect

EWERK DIGITAL GmbH
Brühl 24, D-04109 Leipzig
P +49 341 42649 - 99
F +49 341 42649 - 98
S.Vogel@ewerk.com
www.ewerk.com

Geschäftsführer:
Dr. Erik Wende, Hendrik Schubert, Tassilo Möschke
Registergericht: Leipzig HRB 9065

Zertifiziert nach:
ISO/IEC 27001:2013
DIN EN ISO 9001:2015
DIN ISO/IEC 20000-1:2011

EWERK-Blog | LinkedIn | Xing | Twitter | Facebook

Auskünfte und Angebote per Mail sind freibleibend und unverbindlich.

Re: Managed Storage and HA

Posted by Sven Vogel <S....@ewerk.com>.

Hi Paul, Hi Mike, Hi Syed,

@Paul
> Doesn’t the same problem exist with that? If the agent dies, then the fake volume will stop being updated but the VMs would still be running.

I see different challenges.

@Mike

I don’t know if the „volumes“ table „ready“ state is useful. This will give us a „ready“ state but does not mean its active. I hope I am correct.

@Paul
If this is thinkable where is a best way to implement it that it works. I mean what I need to enable that i know HA is working but lacking on the storage heartbeat.

Cheers Sven

Sven Vogel
Lead Cloud Solution Architect

EWERK DIGITAL GmbH
Brühl 24, D-04109 Leipzig
P +49 341 42649 - 99
F +49 341 42649 - 98
S.Vogel@ewerk.com
www.ewerk.com

Geschäftsführer:
Dr. Erik Wende, Hendrik Schubert, Tassilo Möschke
Registergericht: Leipzig HRB 9065

Zertifiziert nach:
ISO/IEC 27001:2013
DIN EN ISO 9001:2015
DIN ISO/IEC 20000-1:2011

EWERK-Blog | LinkedIn | Xing | Twitter | Facebook

Auskünfte und Angebote per Mail sind freibleibend und unverbindlich.

Re: Managed Storage and HA

Posted by "Tutkowski, Mike" <Mi...@netapp.com>.

Since you know which (SolidFire) volumes are connected to a particular KVM host, I would think you could just use the ListVolumeStats API (described below) from a management server to determine if there is disk activity to any of those volumes.

On 6/8/20, 10:32 AM, "Syed Ahmed" <sa...@cloudops.com.INVALID> wrote:

    NetApp Security WARNING: This is an external email. Do not click links or open attachments unless you recognize the sender and know the content is safe.




    Yeah that's true. I wonder if we could get iscsi information from the
    Solidfire API that way the agent is just responsible for the iscsi login of
    the fake volume and then we just monitor the API for connect time

    On Mon, Jun 8, 2020 at 12:17 PM Paul Angus <pa...@shapeblue.com> wrote:

    > Doesn’t the same problem exist with that?  If the agent dies, then the
    > fake volume will stop being updated but the VMs would still be running.
    >
    > paul.angus@shapeblue.com
    > www.shapeblue.com
    > 3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK
    > @shapeblue
    >
    >
    >
    >
    > -----Original Message-----
    > From: Syed Ahmed <sa...@cloudops.com.INVALID>
    > Sent: 08 June 2020 17:14
    > To: dev@cloudstack.apache.org
    > Subject: Re: Managed Storage and HA
    >
    > My suggestion would be to use a "fake" volume for each host and use that
    > to check the activity of a host. The volume can be updated by the agent
    > periodically and then we can use the above API from the management server
    > to query the volume activity.
    >
    > On Tue, Jun 2, 2020 at 1:18 PM Tutkowski, Mike <Mi...@netapp.com>
    > wrote:
    >
    > > Hi Sven,
    > >
    > > You can use the ListVolumeStats API call (I put in an example request
    > > and response below).
    > >
    > > Since this goes over the management network, though, it's possible if
    > > your management network is down, but your storage network is up that
    > > this call could fail, but your VMs might still have perfectly good
    > > access to their volumes.
    > >
    > > Talk to you later!
    > > Mike
    > >
    > > Request:
    > >
    > > {
    > >    "method": "ListVolumeStats",
    > >    "params": {
    > >         "volumeIDs": [1, 2]
    > >    },
    > >    "id" : 1
    > > }
    > >
    > > Response:
    > >
    > > {
    > >     "id": 1,
    > >     "result": {
    > >         "volumeStats": [
    > >             {
    > >                 "accountID": 1,
    > >                 "actualIOPS": 14,
    > >                 "asyncDelay": null,
    > >                 "averageIOPSize": 13763,
    > >                 "burstIOPSCredit": 0,
    > >                 "clientQueueDepth": 0,
    > >                 "desiredMetadataHosts": null,
    > >                 "latencyUSec": 552,
    > >                 "metadataHosts": {
    > >                     "deadSecondaries": [],
    > >                     "liveSecondaries": [],
    > >                     "primary": 5
    > >                 },
    > >                 "nonZeroBlocks": 10962174,
    > >                 "normalizedIOPS": 34,
    > >                 "readBytes": 747306804224,
    > >                 "readBytesLastSample": 0,
    > >                 "readLatencyUSec": 0,
    > >                 "readLatencyUSecTotal": 11041939920,
    > >                 "readOps": 19877559,
    > >                 "readOpsLastSample": 0,
    > >                 "samplePeriodMSec": 500,
    > >                 "throttle": 0,
    > >                 "timestamp": "2020-06-02T17:14:35.444789Z",
    > >                 "unalignedReads": 2176454,
    > >                 "unalignedWrites": 1438822,
    > >                 "volumeAccessGroups": [
    > >                     1
    > >                 ],
    > >                 "volumeID": 1,
    > >                 "volumeSize": 2147483648000,
    > >                 "volumeUtilization": 0.002266666666666667,
    > >                 "writeBytes": 3231402834432,
    > >                 "writeBytesLastSample": 106496,
    > >                 "writeLatencyUSec": 552,
    > >                 "writeLatencyUSecTotal": 44174792405,
    > >                 "writeOps": 340339085,
    > >                 "writeOpsLastSample": 7,
    > >                 "zeroBlocks": 513325826
    > >             },
    > >             {
    > >                 "accountID": 1,
    > >                 "actualIOPS": 0,
    > >                 "asyncDelay": null,
    > >                 "averageIOPSize": 11261,
    > >                 "burstIOPSCredit": 0,
    > >                 "clientQueueDepth": 0,
    > >                 "desiredMetadataHosts": null,
    > >                 "latencyUSec": 0,
    > >                 "metadataHosts": {
    > >                     "deadSecondaries": [],
    > >                     "liveSecondaries": [],
    > >                     "primary": 5
    > >                 },
    > >                 "nonZeroBlocks": 28816654,
    > >                 "normalizedIOPS": 0,
    > >                 "readBytes": 778768996864,
    > >                 "readBytesLastSample": 0,
    > >                 "readLatencyUSec": 0,
    > >                 "readLatencyUSecTotal": 7068679159,
    > >                 "readOps": 14977610,
    > >                 "readOpsLastSample": 0,
    > >                 "samplePeriodMSec": 500,
    > >                 "throttle": 0,
    > >                 "timestamp": "2020-06-02T17:14:35.445978Z",
    > >                 "unalignedReads": 890959,
    > >                 "unalignedWrites": 358758,
    > >                 "volumeAccessGroups": [
    > >                     1
    > >                 ],
    > >                 "volumeID": 2,
    > >                 "volumeSize": 2147483648000,
    > >                 "volumeUtilization": 0,
    > >                 "writeBytes": 8957684071424,
    > >                 "writeBytesLastSample": 0,
    > >                 "writeLatencyUSec": 0,
    > >                 "writeLatencyUSecTotal": 16780712096,
    > >                 "writeOps": 406101472,
    > >                 "writeOpsLastSample": 0,
    > >                 "zeroBlocks": 495471346
    > >             }
    > >         ]
    > >     }
    > > }
    > >
    > > On 6/2/20, 9:11 AM, "Sven Vogel" <S....@ewerk.com> wrote:
    > >
    > >     NetApp Security WARNING: This is an external email. Do not click
    > > links or open attachments unless you recognize the sender and know the
    > > content is safe.
    > >
    > >
    > >
    > >
    > >     Hi Paul,
    > >
    > >     Thanks for the answer and help.
    > >
    > >     Ok. Secondary Storage is no good solution what I understand.
    > >
    > >     > 1. HAManager
    > >     > 2. HighAvailbilityManager
    > >     > 3. KVMHAConfig
    > >
    > >
    > >     which of the three should we expand and which one should be active?
    > >
    > >     @Mike did you know somethings like that if there is a check of
    > > volume activity?
    > >     Maybe we can poll the API but I think this will be a massive
    > > polling
    > > (overload) if we poll for each volume.
    > >     Ah the moment I don’t have any idea how this could work.
    > >
    > >     Cheers
    > >
    > >     Sven
    > >
    > >
    > >     __
    > >
    > >     Sven Vogel
    > >     Lead Cloud Solution Architect
    > >
    > >     EWERK DIGITAL GmbH
    > >     Brühl 24, D-04109 Leipzig
    > >     P +49 341 42649 - 99
    > >     F +49 341 42649 - 98
    > >     S.Vogel@ewerk.com
    > >     www.ewerk.com
    > >
    > >     Geschäftsführer:
    > >     Dr. Erik Wende, Hendrik Schubert, Tassilo Möschke
    > >     Registergericht: Leipzig HRB 9065
    > >
    > >     Zertifiziert nach:
    > >     ISO/IEC 27001:2013
    > >     DIN EN ISO 9001:2015
    > >     DIN ISO/IEC 20000-1:2011
    > >
    > >     EWERK-Blog | LinkedIn | Xing | Twitter | Facebook
    > >
    > >     Auskünfte und Angebote per Mail sind freibleibend und unverbindlich.
    > >
    > >     Disclaimer Privacy:
    > >     Der Inhalt dieser E-Mail (einschließlich etwaiger beigefügter
    > > Dateien) ist vertraulich und nur für den Empfänger bestimmt. Sollten
    > > Sie nicht der bestimmungsgemäße Empfänger sein, ist Ihnen jegliche
    > > Offenlegung, Vervielfältigung, Weitergabe oder Nutzung des Inhalts
    > > untersagt. Bitte informieren Sie in diesem Fall unverzüglich den
    > > Absender und löschen Sie die E-Mail (einschließlich etwaiger beigefügter
    > Dateien) von Ihrem System.
    > > Vielen Dank.
    > >
    > >     The contents of this e-mail (including any attachments) are
    > > confidential and may be legally privileged. If you are not the
    > > intended recipient of this e-mail, any disclosure, copying,
    > > distribution or use of its contents is strictly prohibited, and you
    > > should please notify the sender immediately and then delete it
    > > (including any attachments) from your system. Thank you.
    > >     > Am 01.06.2020 um 19:30 schrieb Paul Angus
    > > <paul.angus@shapeblue.com
    > > >:
    > >     >
    > >     > Hi Sven,
    > >     >
    > >     > I think that there is a piece of the jigsaw that you are missing.
    > >     >
    > >     > Given that the only thing that we know, is that we can no longer
    > > communicate with the host agent;  To avoid split brain/corruption of
    > > VMs, CloudStack must determine if the guests VMs are still running on
    > > the host not.  The only way we can do that is look for disk activity
    > > created by those VMs.
    > >     >
    > >     > Using a secondary storage heartbeat would give a false 'host is
    > > down' if say a switch went down carrying sec storage and mgmt. traffic
    > >     >
    > >     > Wrt solidfire, you could poll SolidFire via API for activity on
    > > the volumes which belong to the VMs on the unresponsive host.  I don't
    > > know if there is an equivalent for Ceph.
    > >     >
    > >     > Kind regards
    > >     >
    > >     >
    > >     > Paul Angus
    > >     >
    > >     >
    > >     >
    > >     > paul.angus@shapeblue.com
    > >     > www.shapeblue.com
    > >     > 3 London Bridge Street,  3rd floor, News Building, London  SE1
    > 9SGUK
    > >     > @shapeblue
    > >     >
    > >     >
    > >     >
    > >     >
    > >     > -----Original Message-----
    > >     > From: Sven Vogel <S....@ewerk.com>
    > >     > Sent: 01 June 2020 12:30
    > >     > To: dev <de...@cloudstack.apache.org>
    > >     > Subject: Managed Storage and HA
    > >     >
    > >     > Hi Community,
    > >     >
    > >     > I try to encounter how HA works. Our goal is it to make it
    > > usable with managed storage like (Netapp Solidfire / maybe it works
    > > with CEPH too) so if its possible.
    > >     >
    > >     > This is a good guide and for some times we fixed and added the
    > > missing keys.
    > >     >
    > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/High+Availabili
    > > ty+Developer%27s+Guide
    > > <
    > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/High+Availabili
    > > ty+Developer's+Guide
    > > >
    > >     > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Host+HA
    > >     >
    > >     > In the database I found out there are three different types of HA.
    > >     >
    > >     > If you select in the configuration table "SELECT * FROM
    > > `configuration` WHERE `component` LIKE '%ha%' LIMIT 0,1000;“ you will
    > > get three types of components.
    > >     >
    > >     > 1. HAManager
    > >     > 2. HighAvailbilityManager
    > >     > 3. KVMHAConfig
    > >     >
    > >     > "HAManager and HighAvailbilityManager" are the base which was
    > > extended from Rohit with „KVMHAConfig“ - KVM with stonith fencing.
    > >     >
    > >     > I understand all things work together but maybe I need to
    > > understand the process a little bit better.
    > >     >
    > >     >
    > >
    > ------------------------------------------------------------------------------------
    > >     > To clarify I write down what I think to each of them. This is
    > > what I understand but please correct me or help me to understand it a
    > > little bit better.
    > >     >
    > >     > —>I found out that if we use managed storage a restart of
    > > virtual machines only works on the same host. This is what I
    > > understand the lack of the missing heartbeat file on the shared
    > > storage because we don’t have shared storage like NFS.
    > >     >
    > >     > —
    > >     > "If the network ping investigation returns that it cannot detect
    > > the status of the host, CloudStack HA then relies on the hypervisor
    > > specific investigation. For VMware, there is no such investigation as
    > > the hypervisor host handles its own HA. For XenServer and KVM,
    > > CloudStack HA deploys a monitoring script that writes the current
    > > timestamp on to a heartbeat file on shared storage."
    > >     > —
    > >     >
    > >     > And
    > >     >
    > >     > —
    > >     > For the initial release, only KVM with NFS storage will be
    > > supported. However, the storage check component will be implemented in
    > > a modular fashion allowing for checks using other storage platforms(e.g.
    > > Ceph) in the future.
    > >     > —
    > >     >
    > >
    > ------------------------------------------------------------------------------------
    > >     >
    > >     > We would implement a plugin or extend this for managed storage
    > > but at the moment I need to understand where this should happen. Since
    > > managed storage uses different volumes for each VM we should its not
    > > easy to make an storage heartbeat like NFS. the lack of one missing
    > > volume don’t means the hole storage has an problem so I think its not
    > > easy to encounter from one volumes to a complete storage.
    > >     >
    > >     > We don’t use KVMHAConfig a the moment and encounter that if a
    > > Host goes down (offline) the virtual machines will not be restarted on
    > > another host. They will only restarted on the host if the host comes
    > back.(online).
    > > We don’t want a hard fencing of the hosts but we want a correct
    > > determination whether the host is still alive. Fencing would maybe in
    > > our case a little bit hard because we don’t have an hard data
    > > corruption on entire storage.
    > >     >
    > >     > Some questions.
    > >     > 1. let's assume correctly that the HA don’t work without an
    > > shared storage and network ping? Is this the cause why our virtual
    > > machines will not restarted on another host? Is this correct or do we
    > > have an config problem?
    > >     > 2. Where could the plugin be implemented? Is there a preferred
    > place?
    > >     > 3. If point 1. Is correctly I thought the idea would be to add
    > > an global flag to use the secondary storage (NFS) as heartbeat to find
    > > out if there is any host inactive?
    > >     >
    > >     > Thanks and Cheers
    > >     >
    > >     > Sven
    > >     >
    > >     >
    > >     > __
    > >     >
    > >     > Sven Vogel
    > >     > Lead Cloud Solution Architect
    > >     >
    > >     > EWERK DIGITAL GmbH
    > >     > Brühl 24, D-04109 Leipzig
    > >     > P +49 341 42649 - 99
    > >     > F +49 341 42649 - 98
    > >     > S.Vogel@ewerk.com
    > >     > www.ewerk.com
    > >     >
    > >     > Geschäftsführer:
    > >     > Dr. Erik Wende, Hendrik Schubert, Tassilo Möschke
    > >     > Registergericht: Leipzig HRB 9065
    > >     >
    > >     > Support:
    > >     > +49 341 42649 555
    > >     >
    > >     > Zertifiziert nach:
    > >     > ISO/IEC 27001:2013
    > >     > DIN EN ISO 9001:2015
    > >     > DIN ISO/IEC 20000-1:2011
    > >     >
    > >     > ISAE 3402 Typ II Assessed
    > >     >
    > >     > EWERK-Blog<https://blog.ewerk.com/> | LinkedIn<
    > > https://www.linkedin.com/company/ewerk-group> | Xing<
    > > https://www.xing.com/company/ewerk> | Twitter<
    > > https://twitter.com/EWERK_Group> | Facebook<
    > > https://de-de.facebook.com/EWERK.IT/>
    > >     >
    > >     >
    > >     > Auskünfte und Angebote per Mail sind freibleibend und
    > unverbindlich.
    > >     >
    > >     > Disclaimer Privacy:
    > >     > Der Inhalt dieser E-Mail (einschließlich etwaiger beigefügter
    > > Dateien) ist vertraulich und nur für den Empfänger bestimmt. Sollten
    > > Sie nicht der bestimmungsgemäße Empfänger sein, ist Ihnen jegliche
    > > Offenlegung, Vervielfältigung, Weitergabe oder Nutzung des Inhalts
    > > untersagt. Bitte informieren Sie in diesem Fall unverzüglich den
    > > Absender und löschen Sie die E-Mail (einschließlich etwaiger beigefügter
    > Dateien) von Ihrem System.
    > > Vielen Dank.
    > >     >
    > >     > The contents of this e-mail (including any attachments) are
    > > confidential and may be legally privileged. If you are not the
    > > intended recipient of this e-mail, any disclosure, copying,
    > > distribution or use of its contents is strictly prohibited, and you
    > > should please notify the sender immediately and then delete it
    > > (including any attachments) from your system. Thank you.
    > >
    > >
    > >
    >

Re: Managed Storage and HA

Posted by Syed Ahmed <sa...@cloudops.com.INVALID>.

Yeah that's true. I wonder if we could get iscsi information from the
Solidfire API that way the agent is just responsible for the iscsi login of
the fake volume and then we just monitor the API for connect time

On Mon, Jun 8, 2020 at 12:17 PM Paul Angus <pa...@shapeblue.com> wrote:

> Doesn’t the same problem exist with that?  If the agent dies, then the
> fake volume will stop being updated but the VMs would still be running.
>
> paul.angus@shapeblue.com
> www.shapeblue.com
> 3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK
> @shapeblue
>
>
>
>
> -----Original Message-----
> From: Syed Ahmed <sa...@cloudops.com.INVALID>
> Sent: 08 June 2020 17:14
> To: dev@cloudstack.apache.org
> Subject: Re: Managed Storage and HA
>
> My suggestion would be to use a "fake" volume for each host and use that
> to check the activity of a host. The volume can be updated by the agent
> periodically and then we can use the above API from the management server
> to query the volume activity.
>
> On Tue, Jun 2, 2020 at 1:18 PM Tutkowski, Mike <Mi...@netapp.com>
> wrote:
>
> > Hi Sven,
> >
> > You can use the ListVolumeStats API call (I put in an example request
> > and response below).
> >
> > Since this goes over the management network, though, it's possible if
> > your management network is down, but your storage network is up that
> > this call could fail, but your VMs might still have perfectly good
> > access to their volumes.
> >
> > Talk to you later!
> > Mike
> >
> > Request:
> >
> > {
> >    "method": "ListVolumeStats",
> >    "params": {
> >         "volumeIDs": [1, 2]
> >    },
> >    "id" : 1
> > }
> >
> > Response:
> >
> > {
> >     "id": 1,
> >     "result": {
> >         "volumeStats": [
> >             {
> >                 "accountID": 1,
> >                 "actualIOPS": 14,
> >                 "asyncDelay": null,
> >                 "averageIOPSize": 13763,
> >                 "burstIOPSCredit": 0,
> >                 "clientQueueDepth": 0,
> >                 "desiredMetadataHosts": null,
> >                 "latencyUSec": 552,
> >                 "metadataHosts": {
> >                     "deadSecondaries": [],
> >                     "liveSecondaries": [],
> >                     "primary": 5
> >                 },
> >                 "nonZeroBlocks": 10962174,
> >                 "normalizedIOPS": 34,
> >                 "readBytes": 747306804224,
> >                 "readBytesLastSample": 0,
> >                 "readLatencyUSec": 0,
> >                 "readLatencyUSecTotal": 11041939920,
> >                 "readOps": 19877559,
> >                 "readOpsLastSample": 0,
> >                 "samplePeriodMSec": 500,
> >                 "throttle": 0,
> >                 "timestamp": "2020-06-02T17:14:35.444789Z",
> >                 "unalignedReads": 2176454,
> >                 "unalignedWrites": 1438822,
> >                 "volumeAccessGroups": [
> >                     1
> >                 ],
> >                 "volumeID": 1,
> >                 "volumeSize": 2147483648000,
> >                 "volumeUtilization": 0.002266666666666667,
> >                 "writeBytes": 3231402834432,
> >                 "writeBytesLastSample": 106496,
> >                 "writeLatencyUSec": 552,
> >                 "writeLatencyUSecTotal": 44174792405,
> >                 "writeOps": 340339085,
> >                 "writeOpsLastSample": 7,
> >                 "zeroBlocks": 513325826
> >             },
> >             {
> >                 "accountID": 1,
> >                 "actualIOPS": 0,
> >                 "asyncDelay": null,
> >                 "averageIOPSize": 11261,
> >                 "burstIOPSCredit": 0,
> >                 "clientQueueDepth": 0,
> >                 "desiredMetadataHosts": null,
> >                 "latencyUSec": 0,
> >                 "metadataHosts": {
> >                     "deadSecondaries": [],
> >                     "liveSecondaries": [],
> >                     "primary": 5
> >                 },
> >                 "nonZeroBlocks": 28816654,
> >                 "normalizedIOPS": 0,
> >                 "readBytes": 778768996864,
> >                 "readBytesLastSample": 0,
> >                 "readLatencyUSec": 0,
> >                 "readLatencyUSecTotal": 7068679159,
> >                 "readOps": 14977610,
> >                 "readOpsLastSample": 0,
> >                 "samplePeriodMSec": 500,
> >                 "throttle": 0,
> >                 "timestamp": "2020-06-02T17:14:35.445978Z",
> >                 "unalignedReads": 890959,
> >                 "unalignedWrites": 358758,
> >                 "volumeAccessGroups": [
> >                     1
> >                 ],
> >                 "volumeID": 2,
> >                 "volumeSize": 2147483648000,
> >                 "volumeUtilization": 0,
> >                 "writeBytes": 8957684071424,
> >                 "writeBytesLastSample": 0,
> >                 "writeLatencyUSec": 0,
> >                 "writeLatencyUSecTotal": 16780712096,
> >                 "writeOps": 406101472,
> >                 "writeOpsLastSample": 0,
> >                 "zeroBlocks": 495471346
> >             }
> >         ]
> >     }
> > }
> >
> > On 6/2/20, 9:11 AM, "Sven Vogel" <S....@ewerk.com> wrote:
> >
> >     NetApp Security WARNING: This is an external email. Do not click
> > links or open attachments unless you recognize the sender and know the
> > content is safe.
> >
> >
> >
> >
> >     Hi Paul,
> >
> >     Thanks for the answer and help.
> >
> >     Ok. Secondary Storage is no good solution what I understand.
> >
> >     > 1. HAManager
> >     > 2. HighAvailbilityManager
> >     > 3. KVMHAConfig
> >
> >
> >     which of the three should we expand and which one should be active?
> >
> >     @Mike did you know somethings like that if there is a check of
> > volume activity?
> >     Maybe we can poll the API but I think this will be a massive
> > polling
> > (overload) if we poll for each volume.
> >     Ah the moment I don’t have any idea how this could work.
> >
> >     Cheers
> >
> >     Sven
> >
> >
> >     __
> >
> >     Sven Vogel
> >     Lead Cloud Solution Architect
> >
> >     EWERK DIGITAL GmbH
> >     Brühl 24, D-04109 Leipzig
> >     P +49 341 42649 - 99
> >     F +49 341 42649 - 98
> >     S.Vogel@ewerk.com
> >     www.ewerk.com
> >
> >     Geschäftsführer:
> >     Dr. Erik Wende, Hendrik Schubert, Tassilo Möschke
> >     Registergericht: Leipzig HRB 9065
> >
> >     Zertifiziert nach:
> >     ISO/IEC 27001:2013
> >     DIN EN ISO 9001:2015
> >     DIN ISO/IEC 20000-1:2011
> >
> >     EWERK-Blog | LinkedIn | Xing | Twitter | Facebook
> >
> >     Auskünfte und Angebote per Mail sind freibleibend und unverbindlich.
> >
> >     Disclaimer Privacy:
> >     Der Inhalt dieser E-Mail (einschließlich etwaiger beigefügter
> > Dateien) ist vertraulich und nur für den Empfänger bestimmt. Sollten
> > Sie nicht der bestimmungsgemäße Empfänger sein, ist Ihnen jegliche
> > Offenlegung, Vervielfältigung, Weitergabe oder Nutzung des Inhalts
> > untersagt. Bitte informieren Sie in diesem Fall unverzüglich den
> > Absender und löschen Sie die E-Mail (einschließlich etwaiger beigefügter
> Dateien) von Ihrem System.
> > Vielen Dank.
> >
> >     The contents of this e-mail (including any attachments) are
> > confidential and may be legally privileged. If you are not the
> > intended recipient of this e-mail, any disclosure, copying,
> > distribution or use of its contents is strictly prohibited, and you
> > should please notify the sender immediately and then delete it
> > (including any attachments) from your system. Thank you.
> >     > Am 01.06.2020 um 19:30 schrieb Paul Angus
> > <paul.angus@shapeblue.com
> > >:
> >     >
> >     > Hi Sven,
> >     >
> >     > I think that there is a piece of the jigsaw that you are missing.
> >     >
> >     > Given that the only thing that we know, is that we can no longer
> > communicate with the host agent;  To avoid split brain/corruption of
> > VMs, CloudStack must determine if the guests VMs are still running on
> > the host not.  The only way we can do that is look for disk activity
> > created by those VMs.
> >     >
> >     > Using a secondary storage heartbeat would give a false 'host is
> > down' if say a switch went down carrying sec storage and mgmt. traffic
> >     >
> >     > Wrt solidfire, you could poll SolidFire via API for activity on
> > the volumes which belong to the VMs on the unresponsive host.  I don't
> > know if there is an equivalent for Ceph.
> >     >
> >     > Kind regards
> >     >
> >     >
> >     > Paul Angus
> >     >
> >     >
> >     >
> >     > paul.angus@shapeblue.com
> >     > www.shapeblue.com
> >     > 3 London Bridge Street,  3rd floor, News Building, London  SE1
> 9SGUK
> >     > @shapeblue
> >     >
> >     >
> >     >
> >     >
> >     > -----Original Message-----
> >     > From: Sven Vogel <S....@ewerk.com>
> >     > Sent: 01 June 2020 12:30
> >     > To: dev <de...@cloudstack.apache.org>
> >     > Subject: Managed Storage and HA
> >     >
> >     > Hi Community,
> >     >
> >     > I try to encounter how HA works. Our goal is it to make it
> > usable with managed storage like (Netapp Solidfire / maybe it works
> > with CEPH too) so if its possible.
> >     >
> >     > This is a good guide and for some times we fixed and added the
> > missing keys.
> >     >
> > https://cwiki.apache.org/confluence/display/CLOUDSTACK/High+Availabili
> > ty+Developer%27s+Guide
> > <
> > https://cwiki.apache.org/confluence/display/CLOUDSTACK/High+Availabili
> > ty+Developer's+Guide
> > >
> >     > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Host+HA
> >     >
> >     > In the database I found out there are three different types of HA.
> >     >
> >     > If you select in the configuration table "SELECT * FROM
> > `configuration` WHERE `component` LIKE '%ha%' LIMIT 0,1000;“ you will
> > get three types of components.
> >     >
> >     > 1. HAManager
> >     > 2. HighAvailbilityManager
> >     > 3. KVMHAConfig
> >     >
> >     > "HAManager and HighAvailbilityManager" are the base which was
> > extended from Rohit with „KVMHAConfig“ - KVM with stonith fencing.
> >     >
> >     > I understand all things work together but maybe I need to
> > understand the process a little bit better.
> >     >
> >     >
> >
> ------------------------------------------------------------------------------------
> >     > To clarify I write down what I think to each of them. This is
> > what I understand but please correct me or help me to understand it a
> > little bit better.
> >     >
> >     > —>I found out that if we use managed storage a restart of
> > virtual machines only works on the same host. This is what I
> > understand the lack of the missing heartbeat file on the shared
> > storage because we don’t have shared storage like NFS.
> >     >
> >     > —
> >     > "If the network ping investigation returns that it cannot detect
> > the status of the host, CloudStack HA then relies on the hypervisor
> > specific investigation. For VMware, there is no such investigation as
> > the hypervisor host handles its own HA. For XenServer and KVM,
> > CloudStack HA deploys a monitoring script that writes the current
> > timestamp on to a heartbeat file on shared storage."
> >     > —
> >     >
> >     > And
> >     >
> >     > —
> >     > For the initial release, only KVM with NFS storage will be
> > supported. However, the storage check component will be implemented in
> > a modular fashion allowing for checks using other storage platforms(e.g.
> > Ceph) in the future.
> >     > —
> >     >
> >
> ------------------------------------------------------------------------------------
> >     >
> >     > We would implement a plugin or extend this for managed storage
> > but at the moment I need to understand where this should happen. Since
> > managed storage uses different volumes for each VM we should its not
> > easy to make an storage heartbeat like NFS. the lack of one missing
> > volume don’t means the hole storage has an problem so I think its not
> > easy to encounter from one volumes to a complete storage.
> >     >
> >     > We don’t use KVMHAConfig a the moment and encounter that if a
> > Host goes down (offline) the virtual machines will not be restarted on
> > another host. They will only restarted on the host if the host comes
> back.(online).
> > We don’t want a hard fencing of the hosts but we want a correct
> > determination whether the host is still alive. Fencing would maybe in
> > our case a little bit hard because we don’t have an hard data
> > corruption on entire storage.
> >     >
> >     > Some questions.
> >     > 1. let's assume correctly that the HA don’t work without an
> > shared storage and network ping? Is this the cause why our virtual
> > machines will not restarted on another host? Is this correct or do we
> > have an config problem?
> >     > 2. Where could the plugin be implemented? Is there a preferred
> place?
> >     > 3. If point 1. Is correctly I thought the idea would be to add
> > an global flag to use the secondary storage (NFS) as heartbeat to find
> > out if there is any host inactive?
> >     >
> >     > Thanks and Cheers
> >     >
> >     > Sven
> >     >
> >     >
> >     > __
> >     >
> >     > Sven Vogel
> >     > Lead Cloud Solution Architect
> >     >
> >     > EWERK DIGITAL GmbH
> >     > Brühl 24, D-04109 Leipzig
> >     > P +49 341 42649 - 99
> >     > F +49 341 42649 - 98
> >     > S.Vogel@ewerk.com
> >     > www.ewerk.com
> >     >
> >     > Geschäftsführer:
> >     > Dr. Erik Wende, Hendrik Schubert, Tassilo Möschke
> >     > Registergericht: Leipzig HRB 9065
> >     >
> >     > Support:
> >     > +49 341 42649 555
> >     >
> >     > Zertifiziert nach:
> >     > ISO/IEC 27001:2013
> >     > DIN EN ISO 9001:2015
> >     > DIN ISO/IEC 20000-1:2011
> >     >
> >     > ISAE 3402 Typ II Assessed
> >     >
> >     > EWERK-Blog<https://blog.ewerk.com/> | LinkedIn<
> > https://www.linkedin.com/company/ewerk-group> | Xing<
> > https://www.xing.com/company/ewerk> | Twitter<
> > https://twitter.com/EWERK_Group> | Facebook<
> > https://de-de.facebook.com/EWERK.IT/>
> >     >
> >     >
> >     > Auskünfte und Angebote per Mail sind freibleibend und
> unverbindlich.
> >     >
> >     > Disclaimer Privacy:
> >     > Der Inhalt dieser E-Mail (einschließlich etwaiger beigefügter
> > Dateien) ist vertraulich und nur für den Empfänger bestimmt. Sollten
> > Sie nicht der bestimmungsgemäße Empfänger sein, ist Ihnen jegliche
> > Offenlegung, Vervielfältigung, Weitergabe oder Nutzung des Inhalts
> > untersagt. Bitte informieren Sie in diesem Fall unverzüglich den
> > Absender und löschen Sie die E-Mail (einschließlich etwaiger beigefügter
> Dateien) von Ihrem System.
> > Vielen Dank.
> >     >
> >     > The contents of this e-mail (including any attachments) are
> > confidential and may be legally privileged. If you are not the
> > intended recipient of this e-mail, any disclosure, copying,
> > distribution or use of its contents is strictly prohibited, and you
> > should please notify the sender immediately and then delete it
> > (including any attachments) from your system. Thank you.
> >
> >
> >
>

RE: Managed Storage and HA

Posted by Paul Angus <pa...@shapeblue.com>.

Doesn’t the same problem exist with that?  If the agent dies, then the fake volume will stop being updated but the VMs would still be running.

paul.angus@shapeblue.com 
www.shapeblue.com
3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK
@shapeblue
  
 


-----Original Message-----
From: Syed Ahmed <sa...@cloudops.com.INVALID> 
Sent: 08 June 2020 17:14
To: dev@cloudstack.apache.org
Subject: Re: Managed Storage and HA

My suggestion would be to use a "fake" volume for each host and use that to check the activity of a host. The volume can be updated by the agent periodically and then we can use the above API from the management server to query the volume activity.

On Tue, Jun 2, 2020 at 1:18 PM Tutkowski, Mike <Mi...@netapp.com>
wrote:

> Hi Sven,
>
> You can use the ListVolumeStats API call (I put in an example request 
> and response below).
>
> Since this goes over the management network, though, it's possible if 
> your management network is down, but your storage network is up that 
> this call could fail, but your VMs might still have perfectly good 
> access to their volumes.
>
> Talk to you later!
> Mike
>
> Request:
>
> {
>    "method": "ListVolumeStats",
>    "params": {
>         "volumeIDs": [1, 2]
>    },
>    "id" : 1
> }
>
> Response:
>
> {
>     "id": 1,
>     "result": {
>         "volumeStats": [
>             {
>                 "accountID": 1,
>                 "actualIOPS": 14,
>                 "asyncDelay": null,
>                 "averageIOPSize": 13763,
>                 "burstIOPSCredit": 0,
>                 "clientQueueDepth": 0,
>                 "desiredMetadataHosts": null,
>                 "latencyUSec": 552,
>                 "metadataHosts": {
>                     "deadSecondaries": [],
>                     "liveSecondaries": [],
>                     "primary": 5
>                 },
>                 "nonZeroBlocks": 10962174,
>                 "normalizedIOPS": 34,
>                 "readBytes": 747306804224,
>                 "readBytesLastSample": 0,
>                 "readLatencyUSec": 0,
>                 "readLatencyUSecTotal": 11041939920,
>                 "readOps": 19877559,
>                 "readOpsLastSample": 0,
>                 "samplePeriodMSec": 500,
>                 "throttle": 0,
>                 "timestamp": "2020-06-02T17:14:35.444789Z",
>                 "unalignedReads": 2176454,
>                 "unalignedWrites": 1438822,
>                 "volumeAccessGroups": [
>                     1
>                 ],
>                 "volumeID": 1,
>                 "volumeSize": 2147483648000,
>                 "volumeUtilization": 0.002266666666666667,
>                 "writeBytes": 3231402834432,
>                 "writeBytesLastSample": 106496,
>                 "writeLatencyUSec": 552,
>                 "writeLatencyUSecTotal": 44174792405,
>                 "writeOps": 340339085,
>                 "writeOpsLastSample": 7,
>                 "zeroBlocks": 513325826
>             },
>             {
>                 "accountID": 1,
>                 "actualIOPS": 0,
>                 "asyncDelay": null,
>                 "averageIOPSize": 11261,
>                 "burstIOPSCredit": 0,
>                 "clientQueueDepth": 0,
>                 "desiredMetadataHosts": null,
>                 "latencyUSec": 0,
>                 "metadataHosts": {
>                     "deadSecondaries": [],
>                     "liveSecondaries": [],
>                     "primary": 5
>                 },
>                 "nonZeroBlocks": 28816654,
>                 "normalizedIOPS": 0,
>                 "readBytes": 778768996864,
>                 "readBytesLastSample": 0,
>                 "readLatencyUSec": 0,
>                 "readLatencyUSecTotal": 7068679159,
>                 "readOps": 14977610,
>                 "readOpsLastSample": 0,
>                 "samplePeriodMSec": 500,
>                 "throttle": 0,
>                 "timestamp": "2020-06-02T17:14:35.445978Z",
>                 "unalignedReads": 890959,
>                 "unalignedWrites": 358758,
>                 "volumeAccessGroups": [
>                     1
>                 ],
>                 "volumeID": 2,
>                 "volumeSize": 2147483648000,
>                 "volumeUtilization": 0,
>                 "writeBytes": 8957684071424,
>                 "writeBytesLastSample": 0,
>                 "writeLatencyUSec": 0,
>                 "writeLatencyUSecTotal": 16780712096,
>                 "writeOps": 406101472,
>                 "writeOpsLastSample": 0,
>                 "zeroBlocks": 495471346
>             }
>         ]
>     }
> }
>
> On 6/2/20, 9:11 AM, "Sven Vogel" <S....@ewerk.com> wrote:
>
>     NetApp Security WARNING: This is an external email. Do not click 
> links or open attachments unless you recognize the sender and know the 
> content is safe.
>
>
>
>
>     Hi Paul,
>
>     Thanks for the answer and help.
>
>     Ok. Secondary Storage is no good solution what I understand.
>
>     > 1. HAManager
>     > 2. HighAvailbilityManager
>     > 3. KVMHAConfig
>
>
>     which of the three should we expand and which one should be active?
>
>     @Mike did you know somethings like that if there is a check of 
> volume activity?
>     Maybe we can poll the API but I think this will be a massive 
> polling
> (overload) if we poll for each volume.
>     Ah the moment I don’t have any idea how this could work.
>
>     Cheers
>
>     Sven
>
>
>     __
>
>     Sven Vogel
>     Lead Cloud Solution Architect
>
>     EWERK DIGITAL GmbH
>     Brühl 24, D-04109 Leipzig
>     P +49 341 42649 - 99
>     F +49 341 42649 - 98
>     S.Vogel@ewerk.com
>     www.ewerk.com
>
>     Geschäftsführer:
>     Dr. Erik Wende, Hendrik Schubert, Tassilo Möschke
>     Registergericht: Leipzig HRB 9065
>
>     Zertifiziert nach:
>     ISO/IEC 27001:2013
>     DIN EN ISO 9001:2015
>     DIN ISO/IEC 20000-1:2011
>
>     EWERK-Blog | LinkedIn | Xing | Twitter | Facebook
>
>     Auskünfte und Angebote per Mail sind freibleibend und unverbindlich.
>
>     Disclaimer Privacy:
>     Der Inhalt dieser E-Mail (einschließlich etwaiger beigefügter 
> Dateien) ist vertraulich und nur für den Empfänger bestimmt. Sollten 
> Sie nicht der bestimmungsgemäße Empfänger sein, ist Ihnen jegliche 
> Offenlegung, Vervielfältigung, Weitergabe oder Nutzung des Inhalts 
> untersagt. Bitte informieren Sie in diesem Fall unverzüglich den 
> Absender und löschen Sie die E-Mail (einschließlich etwaiger beigefügter Dateien) von Ihrem System.
> Vielen Dank.
>
>     The contents of this e-mail (including any attachments) are 
> confidential and may be legally privileged. If you are not the 
> intended recipient of this e-mail, any disclosure, copying, 
> distribution or use of its contents is strictly prohibited, and you 
> should please notify the sender immediately and then delete it 
> (including any attachments) from your system. Thank you.
>     > Am 01.06.2020 um 19:30 schrieb Paul Angus 
> <paul.angus@shapeblue.com
> >:
>     >
>     > Hi Sven,
>     >
>     > I think that there is a piece of the jigsaw that you are missing.
>     >
>     > Given that the only thing that we know, is that we can no longer 
> communicate with the host agent;  To avoid split brain/corruption of 
> VMs, CloudStack must determine if the guests VMs are still running on 
> the host not.  The only way we can do that is look for disk activity 
> created by those VMs.
>     >
>     > Using a secondary storage heartbeat would give a false 'host is 
> down' if say a switch went down carrying sec storage and mgmt. traffic
>     >
>     > Wrt solidfire, you could poll SolidFire via API for activity on 
> the volumes which belong to the VMs on the unresponsive host.  I don't 
> know if there is an equivalent for Ceph.
>     >
>     > Kind regards
>     >
>     >
>     > Paul Angus
>     >
>     >
>     >
>     > paul.angus@shapeblue.com
>     > www.shapeblue.com
>     > 3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK
>     > @shapeblue
>     >
>     >
>     >
>     >
>     > -----Original Message-----
>     > From: Sven Vogel <S....@ewerk.com>
>     > Sent: 01 June 2020 12:30
>     > To: dev <de...@cloudstack.apache.org>
>     > Subject: Managed Storage and HA
>     >
>     > Hi Community,
>     >
>     > I try to encounter how HA works. Our goal is it to make it 
> usable with managed storage like (Netapp Solidfire / maybe it works 
> with CEPH too) so if its possible.
>     >
>     > This is a good guide and for some times we fixed and added the 
> missing keys.
>     >
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/High+Availabili
> ty+Developer%27s+Guide
> <
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/High+Availabili
> ty+Developer's+Guide
> >
>     > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Host+HA
>     >
>     > In the database I found out there are three different types of HA.
>     >
>     > If you select in the configuration table "SELECT * FROM 
> `configuration` WHERE `component` LIKE '%ha%' LIMIT 0,1000;“ you will 
> get three types of components.
>     >
>     > 1. HAManager
>     > 2. HighAvailbilityManager
>     > 3. KVMHAConfig
>     >
>     > "HAManager and HighAvailbilityManager" are the base which was 
> extended from Rohit with „KVMHAConfig“ - KVM with stonith fencing.
>     >
>     > I understand all things work together but maybe I need to 
> understand the process a little bit better.
>     >
>     >
> ------------------------------------------------------------------------------------
>     > To clarify I write down what I think to each of them. This is 
> what I understand but please correct me or help me to understand it a 
> little bit better.
>     >
>     > —>I found out that if we use managed storage a restart of 
> virtual machines only works on the same host. This is what I 
> understand the lack of the missing heartbeat file on the shared 
> storage because we don’t have shared storage like NFS.
>     >
>     > —
>     > "If the network ping investigation returns that it cannot detect 
> the status of the host, CloudStack HA then relies on the hypervisor 
> specific investigation. For VMware, there is no such investigation as 
> the hypervisor host handles its own HA. For XenServer and KVM, 
> CloudStack HA deploys a monitoring script that writes the current 
> timestamp on to a heartbeat file on shared storage."
>     > —
>     >
>     > And
>     >
>     > —
>     > For the initial release, only KVM with NFS storage will be 
> supported. However, the storage check component will be implemented in 
> a modular fashion allowing for checks using other storage platforms(e.g.
> Ceph) in the future.
>     > —
>     >
> ------------------------------------------------------------------------------------
>     >
>     > We would implement a plugin or extend this for managed storage 
> but at the moment I need to understand where this should happen. Since 
> managed storage uses different volumes for each VM we should its not 
> easy to make an storage heartbeat like NFS. the lack of one missing 
> volume don’t means the hole storage has an problem so I think its not 
> easy to encounter from one volumes to a complete storage.
>     >
>     > We don’t use KVMHAConfig a the moment and encounter that if a 
> Host goes down (offline) the virtual machines will not be restarted on 
> another host. They will only restarted on the host if the host comes back.(online).
> We don’t want a hard fencing of the hosts but we want a correct 
> determination whether the host is still alive. Fencing would maybe in 
> our case a little bit hard because we don’t have an hard data 
> corruption on entire storage.
>     >
>     > Some questions.
>     > 1. let's assume correctly that the HA don’t work without an 
> shared storage and network ping? Is this the cause why our virtual 
> machines will not restarted on another host? Is this correct or do we 
> have an config problem?
>     > 2. Where could the plugin be implemented? Is there a preferred place?
>     > 3. If point 1. Is correctly I thought the idea would be to add 
> an global flag to use the secondary storage (NFS) as heartbeat to find 
> out if there is any host inactive?
>     >
>     > Thanks and Cheers
>     >
>     > Sven
>     >
>     >
>     > __
>     >
>     > Sven Vogel
>     > Lead Cloud Solution Architect
>     >
>     > EWERK DIGITAL GmbH
>     > Brühl 24, D-04109 Leipzig
>     > P +49 341 42649 - 99
>     > F +49 341 42649 - 98
>     > S.Vogel@ewerk.com
>     > www.ewerk.com
>     >
>     > Geschäftsführer:
>     > Dr. Erik Wende, Hendrik Schubert, Tassilo Möschke
>     > Registergericht: Leipzig HRB 9065
>     >
>     > Support:
>     > +49 341 42649 555
>     >
>     > Zertifiziert nach:
>     > ISO/IEC 27001:2013
>     > DIN EN ISO 9001:2015
>     > DIN ISO/IEC 20000-1:2011
>     >
>     > ISAE 3402 Typ II Assessed
>     >
>     > EWERK-Blog<https://blog.ewerk.com/> | LinkedIn< 
> https://www.linkedin.com/company/ewerk-group> | Xing< 
> https://www.xing.com/company/ewerk> | Twitter< 
> https://twitter.com/EWERK_Group> | Facebook< 
> https://de-de.facebook.com/EWERK.IT/>
>     >
>     >
>     > Auskünfte und Angebote per Mail sind freibleibend und unverbindlich.
>     >
>     > Disclaimer Privacy:
>     > Der Inhalt dieser E-Mail (einschließlich etwaiger beigefügter
> Dateien) ist vertraulich und nur für den Empfänger bestimmt. Sollten 
> Sie nicht der bestimmungsgemäße Empfänger sein, ist Ihnen jegliche 
> Offenlegung, Vervielfältigung, Weitergabe oder Nutzung des Inhalts 
> untersagt. Bitte informieren Sie in diesem Fall unverzüglich den 
> Absender und löschen Sie die E-Mail (einschließlich etwaiger beigefügter Dateien) von Ihrem System.
> Vielen Dank.
>     >
>     > The contents of this e-mail (including any attachments) are 
> confidential and may be legally privileged. If you are not the 
> intended recipient of this e-mail, any disclosure, copying, 
> distribution or use of its contents is strictly prohibited, and you 
> should please notify the sender immediately and then delete it 
> (including any attachments) from your system. Thank you.
>
>
>

Re: Managed Storage and HA

Posted by Syed Ahmed <sa...@cloudops.com.INVALID>.

My suggestion would be to use a "fake" volume for each host and use that to
check the activity of a host. The volume can be updated by the agent
periodically and then we can use the above API from the management server
to query the volume activity.

On Tue, Jun 2, 2020 at 1:18 PM Tutkowski, Mike <Mi...@netapp.com>
wrote:

> Hi Sven,
>
> You can use the ListVolumeStats API call (I put in an example request and
> response below).
>
> Since this goes over the management network, though, it's possible if your
> management network is down, but your storage network is up that this call
> could fail, but your VMs might still have perfectly good access to their
> volumes.
>
> Talk to you later!
> Mike
>
> Request:
>
> {
>    "method": "ListVolumeStats",
>    "params": {
>         "volumeIDs": [1, 2]
>    },
>    "id" : 1
> }
>
> Response:
>
> {
>     "id": 1,
>     "result": {
>         "volumeStats": [
>             {
>                 "accountID": 1,
>                 "actualIOPS": 14,
>                 "asyncDelay": null,
>                 "averageIOPSize": 13763,
>                 "burstIOPSCredit": 0,
>                 "clientQueueDepth": 0,
>                 "desiredMetadataHosts": null,
>                 "latencyUSec": 552,
>                 "metadataHosts": {
>                     "deadSecondaries": [],
>                     "liveSecondaries": [],
>                     "primary": 5
>                 },
>                 "nonZeroBlocks": 10962174,
>                 "normalizedIOPS": 34,
>                 "readBytes": 747306804224,
>                 "readBytesLastSample": 0,
>                 "readLatencyUSec": 0,
>                 "readLatencyUSecTotal": 11041939920,
>                 "readOps": 19877559,
>                 "readOpsLastSample": 0,
>                 "samplePeriodMSec": 500,
>                 "throttle": 0,
>                 "timestamp": "2020-06-02T17:14:35.444789Z",
>                 "unalignedReads": 2176454,
>                 "unalignedWrites": 1438822,
>                 "volumeAccessGroups": [
>                     1
>                 ],
>                 "volumeID": 1,
>                 "volumeSize": 2147483648000,
>                 "volumeUtilization": 0.002266666666666667,
>                 "writeBytes": 3231402834432,
>                 "writeBytesLastSample": 106496,
>                 "writeLatencyUSec": 552,
>                 "writeLatencyUSecTotal": 44174792405,
>                 "writeOps": 340339085,
>                 "writeOpsLastSample": 7,
>                 "zeroBlocks": 513325826
>             },
>             {
>                 "accountID": 1,
>                 "actualIOPS": 0,
>                 "asyncDelay": null,
>                 "averageIOPSize": 11261,
>                 "burstIOPSCredit": 0,
>                 "clientQueueDepth": 0,
>                 "desiredMetadataHosts": null,
>                 "latencyUSec": 0,
>                 "metadataHosts": {
>                     "deadSecondaries": [],
>                     "liveSecondaries": [],
>                     "primary": 5
>                 },
>                 "nonZeroBlocks": 28816654,
>                 "normalizedIOPS": 0,
>                 "readBytes": 778768996864,
>                 "readBytesLastSample": 0,
>                 "readLatencyUSec": 0,
>                 "readLatencyUSecTotal": 7068679159,
>                 "readOps": 14977610,
>                 "readOpsLastSample": 0,
>                 "samplePeriodMSec": 500,
>                 "throttle": 0,
>                 "timestamp": "2020-06-02T17:14:35.445978Z",
>                 "unalignedReads": 890959,
>                 "unalignedWrites": 358758,
>                 "volumeAccessGroups": [
>                     1
>                 ],
>                 "volumeID": 2,
>                 "volumeSize": 2147483648000,
>                 "volumeUtilization": 0,
>                 "writeBytes": 8957684071424,
>                 "writeBytesLastSample": 0,
>                 "writeLatencyUSec": 0,
>                 "writeLatencyUSecTotal": 16780712096,
>                 "writeOps": 406101472,
>                 "writeOpsLastSample": 0,
>                 "zeroBlocks": 495471346
>             }
>         ]
>     }
> }
>
> On 6/2/20, 9:11 AM, "Sven Vogel" <S....@ewerk.com> wrote:
>
>     NetApp Security WARNING: This is an external email. Do not click links
> or open attachments unless you recognize the sender and know the content is
> safe.
>
>
>
>
>     Hi Paul,
>
>     Thanks for the answer and help.
>
>     Ok. Secondary Storage is no good solution what I understand.
>
>     > 1. HAManager
>     > 2. HighAvailbilityManager
>     > 3. KVMHAConfig
>
>
>     which of the three should we expand and which one should be active?
>
>     @Mike did you know somethings like that if there is a check of volume
> activity?
>     Maybe we can poll the API but I think this will be a massive polling
> (overload) if we poll for each volume.
>     Ah the moment I don’t have any idea how this could work.
>
>     Cheers
>
>     Sven
>
>
>     __
>
>     Sven Vogel
>     Lead Cloud Solution Architect
>
>     EWERK DIGITAL GmbH
>     Brühl 24, D-04109 Leipzig
>     P +49 341 42649 - 99
>     F +49 341 42649 - 98
>     S.Vogel@ewerk.com
>     www.ewerk.com
>
>     Geschäftsführer:
>     Dr. Erik Wende, Hendrik Schubert, Tassilo Möschke
>     Registergericht: Leipzig HRB 9065
>
>     Zertifiziert nach:
>     ISO/IEC 27001:2013
>     DIN EN ISO 9001:2015
>     DIN ISO/IEC 20000-1:2011
>
>     EWERK-Blog | LinkedIn | Xing | Twitter | Facebook
>
>     Auskünfte und Angebote per Mail sind freibleibend und unverbindlich.
>
>     Disclaimer Privacy:
>     Der Inhalt dieser E-Mail (einschließlich etwaiger beigefügter Dateien)
> ist vertraulich und nur für den Empfänger bestimmt. Sollten Sie nicht der
> bestimmungsgemäße Empfänger sein, ist Ihnen jegliche Offenlegung,
> Vervielfältigung, Weitergabe oder Nutzung des Inhalts untersagt. Bitte
> informieren Sie in diesem Fall unverzüglich den Absender und löschen Sie
> die E-Mail (einschließlich etwaiger beigefügter Dateien) von Ihrem System.
> Vielen Dank.
>
>     The contents of this e-mail (including any attachments) are
> confidential and may be legally privileged. If you are not the intended
> recipient of this e-mail, any disclosure, copying, distribution or use of
> its contents is strictly prohibited, and you should please notify the
> sender immediately and then delete it (including any attachments) from your
> system. Thank you.
>     > Am 01.06.2020 um 19:30 schrieb Paul Angus <paul.angus@shapeblue.com
> >:
>     >
>     > Hi Sven,
>     >
>     > I think that there is a piece of the jigsaw that you are missing.
>     >
>     > Given that the only thing that we know, is that we can no longer
> communicate with the host agent;  To avoid split brain/corruption of VMs,
> CloudStack must determine if the guests VMs are still running on the host
> not.  The only way we can do that is look for disk activity created by
> those VMs.
>     >
>     > Using a secondary storage heartbeat would give a false 'host is
> down' if say a switch went down carrying sec storage and mgmt. traffic
>     >
>     > Wrt solidfire, you could poll SolidFire via API for activity on the
> volumes which belong to the VMs on the unresponsive host.  I don't know if
> there is an equivalent for Ceph.
>     >
>     > Kind regards
>     >
>     >
>     > Paul Angus
>     >
>     >
>     >
>     > paul.angus@shapeblue.com
>     > www.shapeblue.com
>     > 3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK
>     > @shapeblue
>     >
>     >
>     >
>     >
>     > -----Original Message-----
>     > From: Sven Vogel <S....@ewerk.com>
>     > Sent: 01 June 2020 12:30
>     > To: dev <de...@cloudstack.apache.org>
>     > Subject: Managed Storage and HA
>     >
>     > Hi Community,
>     >
>     > I try to encounter how HA works. Our goal is it to make it usable
> with managed storage like (Netapp Solidfire / maybe it works with CEPH too)
> so if its possible.
>     >
>     > This is a good guide and for some times we fixed and added the
> missing keys.
>     >
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/High+Availability+Developer%27s+Guide
> <
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/High+Availability+Developer's+Guide
> >
>     > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Host+HA
>     >
>     > In the database I found out there are three different types of HA.
>     >
>     > If you select in the configuration table "SELECT * FROM
> `configuration` WHERE `component` LIKE '%ha%' LIMIT 0,1000;“ you will get
> three types of components.
>     >
>     > 1. HAManager
>     > 2. HighAvailbilityManager
>     > 3. KVMHAConfig
>     >
>     > "HAManager and HighAvailbilityManager" are the base which was
> extended from Rohit with „KVMHAConfig“ - KVM with stonith fencing.
>     >
>     > I understand all things work together but maybe I need to understand
> the process a little bit better.
>     >
>     >
> ------------------------------------------------------------------------------------
>     > To clarify I write down what I think to each of them. This is what I
> understand but please correct me or help me to understand it a little bit
> better.
>     >
>     > —>I found out that if we use managed storage a restart of virtual
> machines only works on the same host. This is what I understand the lack of
> the missing heartbeat file on the shared storage because we don’t have
> shared storage like NFS.
>     >
>     > —
>     > "If the network ping investigation returns that it cannot detect the
> status of the host, CloudStack HA then relies on the hypervisor specific
> investigation. For VMware, there is no such investigation as the hypervisor
> host handles its own HA. For XenServer and KVM, CloudStack HA deploys a
> monitoring script that writes the current timestamp on to a heartbeat file
> on shared storage."
>     > —
>     >
>     > And
>     >
>     > —
>     > For the initial release, only KVM with NFS storage will be
> supported. However, the storage check component will be implemented in a
> modular fashion allowing for checks using other storage platforms(e.g.
> Ceph) in the future.
>     > —
>     >
> ------------------------------------------------------------------------------------
>     >
>     > We would implement a plugin or extend this for managed storage but
> at the moment I need to understand where this should happen. Since managed
> storage uses different volumes for each VM we should its not easy to make
> an storage heartbeat like NFS. the lack of one missing volume don’t means
> the hole storage has an problem so I think its not easy to encounter from
> one volumes to a complete storage.
>     >
>     > We don’t use KVMHAConfig a the moment and encounter that if a Host
> goes down (offline) the virtual machines will not be restarted on another
> host. They will only restarted on the host if the host comes back.(online).
> We don’t want a hard fencing of the hosts but we want a correct
> determination whether the host is still alive. Fencing would maybe in our
> case a little bit hard because we don’t have an hard data corruption on
> entire storage.
>     >
>     > Some questions.
>     > 1. let's assume correctly that the HA don’t work without an shared
> storage and network ping? Is this the cause why our virtual machines will
> not restarted on another host? Is this correct or do we have an config
> problem?
>     > 2. Where could the plugin be implemented? Is there a preferred place?
>     > 3. If point 1. Is correctly I thought the idea would be to add an
> global flag to use the secondary storage (NFS) as heartbeat to find out if
> there is any host inactive?
>     >
>     > Thanks and Cheers
>     >
>     > Sven
>     >
>     >
>     > __
>     >
>     > Sven Vogel
>     > Lead Cloud Solution Architect
>     >
>     > EWERK DIGITAL GmbH
>     > Brühl 24, D-04109 Leipzig
>     > P +49 341 42649 - 99
>     > F +49 341 42649 - 98
>     > S.Vogel@ewerk.com
>     > www.ewerk.com
>     >
>     > Geschäftsführer:
>     > Dr. Erik Wende, Hendrik Schubert, Tassilo Möschke
>     > Registergericht: Leipzig HRB 9065
>     >
>     > Support:
>     > +49 341 42649 555
>     >
>     > Zertifiziert nach:
>     > ISO/IEC 27001:2013
>     > DIN EN ISO 9001:2015
>     > DIN ISO/IEC 20000-1:2011
>     >
>     > ISAE 3402 Typ II Assessed
>     >
>     > EWERK-Blog<https://blog.ewerk.com/> | LinkedIn<
> https://www.linkedin.com/company/ewerk-group> | Xing<
> https://www.xing.com/company/ewerk> | Twitter<
> https://twitter.com/EWERK_Group> | Facebook<
> https://de-de.facebook.com/EWERK.IT/>
>     >
>     >
>     > Auskünfte und Angebote per Mail sind freibleibend und unverbindlich.
>     >
>     > Disclaimer Privacy:
>     > Der Inhalt dieser E-Mail (einschließlich etwaiger beigefügter
> Dateien) ist vertraulich und nur für den Empfänger bestimmt. Sollten Sie
> nicht der bestimmungsgemäße Empfänger sein, ist Ihnen jegliche Offenlegung,
> Vervielfältigung, Weitergabe oder Nutzung des Inhalts untersagt. Bitte
> informieren Sie in diesem Fall unverzüglich den Absender und löschen Sie
> die E-Mail (einschließlich etwaiger beigefügter Dateien) von Ihrem System.
> Vielen Dank.
>     >
>     > The contents of this e-mail (including any attachments) are
> confidential and may be legally privileged. If you are not the intended
> recipient of this e-mail, any disclosure, copying, distribution or use of
> its contents is strictly prohibited, and you should please notify the
> sender immediately and then delete it (including any attachments) from your
> system. Thank you.
>
>
>

Re: Managed Storage and HA

Posted by "Tutkowski, Mike" <Mi...@netapp.com>.

Hi Sven,

You can use the ListVolumeStats API call (I put in an example request and response below).

Since this goes over the management network, though, it's possible if your management network is down, but your storage network is up that this call could fail, but your VMs might still have perfectly good access to their volumes.

Talk to you later!
Mike

Request:

{
   "method": "ListVolumeStats",
   "params": {
   	"volumeIDs": [1, 2]
   },
   "id" : 1
}

Response:

{
    "id": 1,
    "result": {
        "volumeStats": [
            {
                "accountID": 1,
                "actualIOPS": 14,
                "asyncDelay": null,
                "averageIOPSize": 13763,
                "burstIOPSCredit": 0,
                "clientQueueDepth": 0,
                "desiredMetadataHosts": null,
                "latencyUSec": 552,
                "metadataHosts": {
                    "deadSecondaries": [],
                    "liveSecondaries": [],
                    "primary": 5
                },
                "nonZeroBlocks": 10962174,
                "normalizedIOPS": 34,
                "readBytes": 747306804224,
                "readBytesLastSample": 0,
                "readLatencyUSec": 0,
                "readLatencyUSecTotal": 11041939920,
                "readOps": 19877559,
                "readOpsLastSample": 0,
                "samplePeriodMSec": 500,
                "throttle": 0,
                "timestamp": "2020-06-02T17:14:35.444789Z",
                "unalignedReads": 2176454,
                "unalignedWrites": 1438822,
                "volumeAccessGroups": [
                    1
                ],
                "volumeID": 1,
                "volumeSize": 2147483648000,
                "volumeUtilization": 0.002266666666666667,
                "writeBytes": 3231402834432,
                "writeBytesLastSample": 106496,
                "writeLatencyUSec": 552,
                "writeLatencyUSecTotal": 44174792405,
                "writeOps": 340339085,
                "writeOpsLastSample": 7,
                "zeroBlocks": 513325826
            },
            {
                "accountID": 1,
                "actualIOPS": 0,
                "asyncDelay": null,
                "averageIOPSize": 11261,
                "burstIOPSCredit": 0,
                "clientQueueDepth": 0,
                "desiredMetadataHosts": null,
                "latencyUSec": 0,
                "metadataHosts": {
                    "deadSecondaries": [],
                    "liveSecondaries": [],
                    "primary": 5
                },
                "nonZeroBlocks": 28816654,
                "normalizedIOPS": 0,
                "readBytes": 778768996864,
                "readBytesLastSample": 0,
                "readLatencyUSec": 0,
                "readLatencyUSecTotal": 7068679159,
                "readOps": 14977610,
                "readOpsLastSample": 0,
                "samplePeriodMSec": 500,
                "throttle": 0,
                "timestamp": "2020-06-02T17:14:35.445978Z",
                "unalignedReads": 890959,
                "unalignedWrites": 358758,
                "volumeAccessGroups": [
                    1
                ],
                "volumeID": 2,
                "volumeSize": 2147483648000,
                "volumeUtilization": 0,
                "writeBytes": 8957684071424,
                "writeBytesLastSample": 0,
                "writeLatencyUSec": 0,
                "writeLatencyUSecTotal": 16780712096,
                "writeOps": 406101472,
                "writeOpsLastSample": 0,
                "zeroBlocks": 495471346
            }
        ]
    }
}

On 6/2/20, 9:11 AM, "Sven Vogel" <S....@ewerk.com> wrote:

    NetApp Security WARNING: This is an external email. Do not click links or open attachments unless you recognize the sender and know the content is safe.




    Hi Paul,

    Thanks for the answer and help.

    Ok. Secondary Storage is no good solution what I understand.

    > 1. HAManager
    > 2. HighAvailbilityManager
    > 3. KVMHAConfig


    which of the three should we expand and which one should be active?

    @Mike did you know somethings like that if there is a check of volume activity?
    Maybe we can poll the API but I think this will be a massive polling (overload) if we poll for each volume.
    Ah the moment I don’t have any idea how this could work.

    Cheers

    Sven


    __

    Sven Vogel
    Lead Cloud Solution Architect

    EWERK DIGITAL GmbH
    Brühl 24, D-04109 Leipzig
    P +49 341 42649 - 99
    F +49 341 42649 - 98
    S.Vogel@ewerk.com
    www.ewerk.com

    Geschäftsführer:
    Dr. Erik Wende, Hendrik Schubert, Tassilo Möschke
    Registergericht: Leipzig HRB 9065

    Zertifiziert nach:
    ISO/IEC 27001:2013
    DIN EN ISO 9001:2015
    DIN ISO/IEC 20000-1:2011

    EWERK-Blog | LinkedIn | Xing | Twitter | Facebook

    Auskünfte und Angebote per Mail sind freibleibend und unverbindlich.

    Disclaimer Privacy:
    Der Inhalt dieser E-Mail (einschließlich etwaiger beigefügter Dateien) ist vertraulich und nur für den Empfänger bestimmt. Sollten Sie nicht der bestimmungsgemäße Empfänger sein, ist Ihnen jegliche Offenlegung, Vervielfältigung, Weitergabe oder Nutzung des Inhalts untersagt. Bitte informieren Sie in diesem Fall unverzüglich den Absender und löschen Sie die E-Mail (einschließlich etwaiger beigefügter Dateien) von Ihrem System. Vielen Dank.

    The contents of this e-mail (including any attachments) are confidential and may be legally privileged. If you are not the intended recipient of this e-mail, any disclosure, copying, distribution or use of its contents is strictly prohibited, and you should please notify the sender immediately and then delete it (including any attachments) from your system. Thank you.
    > Am 01.06.2020 um 19:30 schrieb Paul Angus <pa...@shapeblue.com>:
    >
    > Hi Sven,
    >
    > I think that there is a piece of the jigsaw that you are missing.
    >
    > Given that the only thing that we know, is that we can no longer communicate with the host agent;  To avoid split brain/corruption of VMs, CloudStack must determine if the guests VMs are still running on the host not.  The only way we can do that is look for disk activity created by those VMs.
    >
    > Using a secondary storage heartbeat would give a false 'host is down' if say a switch went down carrying sec storage and mgmt. traffic
    >
    > Wrt solidfire, you could poll SolidFire via API for activity on the volumes which belong to the VMs on the unresponsive host.  I don't know if there is an equivalent for Ceph.
    >
    > Kind regards
    >
    >
    > Paul Angus
    >
    >
    >
    > paul.angus@shapeblue.com
    > www.shapeblue.com
    > 3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK
    > @shapeblue
    >
    >
    >
    >
    > -----Original Message-----
    > From: Sven Vogel <S....@ewerk.com>
    > Sent: 01 June 2020 12:30
    > To: dev <de...@cloudstack.apache.org>
    > Subject: Managed Storage and HA
    >
    > Hi Community,
    >
    > I try to encounter how HA works. Our goal is it to make it usable with managed storage like (Netapp Solidfire / maybe it works with CEPH too) so if its possible.
    >
    > This is a good guide and for some times we fixed and added the missing keys.
    > https://cwiki.apache.org/confluence/display/CLOUDSTACK/High+Availability+Developer%27s+Guide<https://cwiki.apache.org/confluence/display/CLOUDSTACK/High+Availability+Developer's+Guide>
    > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Host+HA
    >
    > In the database I found out there are three different types of HA.
    >
    > If you select in the configuration table "SELECT * FROM `configuration` WHERE `component` LIKE '%ha%' LIMIT 0,1000;“ you will get three types of components.
    >
    > 1. HAManager
    > 2. HighAvailbilityManager
    > 3. KVMHAConfig
    >
    > "HAManager and HighAvailbilityManager" are the base which was extended from Rohit with „KVMHAConfig“ - KVM with stonith fencing.
    >
    > I understand all things work together but maybe I need to understand the process a little bit better.
    >
    > ------------------------------------------------------------------------------------
    > To clarify I write down what I think to each of them. This is what I understand but please correct me or help me to understand it a little bit better.
    >
    > —>I found out that if we use managed storage a restart of virtual machines only works on the same host. This is what I understand the lack of the missing heartbeat file on the shared storage because we don’t have shared storage like NFS.
    >
    > —
    > "If the network ping investigation returns that it cannot detect the status of the host, CloudStack HA then relies on the hypervisor specific investigation. For VMware, there is no such investigation as the hypervisor host handles its own HA. For XenServer and KVM, CloudStack HA deploys a monitoring script that writes the current timestamp on to a heartbeat file on shared storage."
    > —
    >
    > And
    >
    > —
    > For the initial release, only KVM with NFS storage will be supported. However, the storage check component will be implemented in a modular fashion allowing for checks using other storage platforms(e.g. Ceph) in the future.
    > —
    > ------------------------------------------------------------------------------------
    >
    > We would implement a plugin or extend this for managed storage but at the moment I need to understand where this should happen. Since managed storage uses different volumes for each VM we should its not easy to make an storage heartbeat like NFS. the lack of one missing volume don’t means the hole storage has an problem so I think its not easy to encounter from one volumes to a complete storage.
    >
    > We don’t use KVMHAConfig a the moment and encounter that if a Host goes down (offline) the virtual machines will not be restarted on another host. They will only restarted on the host if the host comes back.(online). We don’t want a hard fencing of the hosts but we want a correct determination whether the host is still alive. Fencing would maybe in our case a little bit hard because we don’t have an hard data corruption on entire storage.
    >
    > Some questions.
    > 1. let's assume correctly that the HA don’t work without an shared storage and network ping? Is this the cause why our virtual machines will not restarted on another host? Is this correct or do we have an config problem?
    > 2. Where could the plugin be implemented? Is there a preferred place?
    > 3. If point 1. Is correctly I thought the idea would be to add an global flag to use the secondary storage (NFS) as heartbeat to find out if there is any host inactive?
    >
    > Thanks and Cheers
    >
    > Sven
    >
    >
    > __
    >
    > Sven Vogel
    > Lead Cloud Solution Architect
    >
    > EWERK DIGITAL GmbH
    > Brühl 24, D-04109 Leipzig
    > P +49 341 42649 - 99
    > F +49 341 42649 - 98
    > S.Vogel@ewerk.com
    > www.ewerk.com
    >
    > Geschäftsführer:
    > Dr. Erik Wende, Hendrik Schubert, Tassilo Möschke
    > Registergericht: Leipzig HRB 9065
    >
    > Support:
    > +49 341 42649 555
    >
    > Zertifiziert nach:
    > ISO/IEC 27001:2013
    > DIN EN ISO 9001:2015
    > DIN ISO/IEC 20000-1:2011
    >
    > ISAE 3402 Typ II Assessed
    >
    > EWERK-Blog<https://blog.ewerk.com/> | LinkedIn<https://www.linkedin.com/company/ewerk-group> | Xing<https://www.xing.com/company/ewerk> | Twitter<https://twitter.com/EWERK_Group> | Facebook<https://de-de.facebook.com/EWERK.IT/>
    >
    >
    > Auskünfte und Angebote per Mail sind freibleibend und unverbindlich.
    >
    > Disclaimer Privacy:
    > Der Inhalt dieser E-Mail (einschließlich etwaiger beigefügter Dateien) ist vertraulich und nur für den Empfänger bestimmt. Sollten Sie nicht der bestimmungsgemäße Empfänger sein, ist Ihnen jegliche Offenlegung, Vervielfältigung, Weitergabe oder Nutzung des Inhalts untersagt. Bitte informieren Sie in diesem Fall unverzüglich den Absender und löschen Sie die E-Mail (einschließlich etwaiger beigefügter Dateien) von Ihrem System. Vielen Dank.
    >
    > The contents of this e-mail (including any attachments) are confidential and may be legally privileged. If you are not the intended recipient of this e-mail, any disclosure, copying, distribution or use of its contents is strictly prohibited, and you should please notify the sender immediately and then delete it (including any attachments) from your system. Thank you.

Re: Managed Storage and HA

Posted by Sven Vogel <S....@ewerk.com>.

Hi Paul,

Thanks for the answer and help.

Ok. Secondary Storage is no good solution what I understand.

> 1. HAManager
> 2. HighAvailbilityManager
> 3. KVMHAConfig


which of the three should we expand and which one should be active?

@Mike did you know somethings like that if there is a check of volume activity?
Maybe we can poll the API but I think this will be a massive polling (overload) if we poll for each volume.
Ah the moment I don’t have any idea how this could work.

Cheers

Sven


__

Sven Vogel
Lead Cloud Solution Architect

EWERK DIGITAL GmbH
Brühl 24, D-04109 Leipzig
P +49 341 42649 - 99
F +49 341 42649 - 98
S.Vogel@ewerk.com
www.ewerk.com

Geschäftsführer:
Dr. Erik Wende, Hendrik Schubert, Tassilo Möschke
Registergericht: Leipzig HRB 9065

Zertifiziert nach:
ISO/IEC 27001:2013
DIN EN ISO 9001:2015
DIN ISO/IEC 20000-1:2011

EWERK-Blog | LinkedIn | Xing | Twitter | Facebook

Auskünfte und Angebote per Mail sind freibleibend und unverbindlich.

Disclaimer Privacy:
Der Inhalt dieser E-Mail (einschließlich etwaiger beigefügter Dateien) ist vertraulich und nur für den Empfänger bestimmt. Sollten Sie nicht der bestimmungsgemäße Empfänger sein, ist Ihnen jegliche Offenlegung, Vervielfältigung, Weitergabe oder Nutzung des Inhalts untersagt. Bitte informieren Sie in diesem Fall unverzüglich den Absender und löschen Sie die E-Mail (einschließlich etwaiger beigefügter Dateien) von Ihrem System. Vielen Dank.

The contents of this e-mail (including any attachments) are confidential and may be legally privileged. If you are not the intended recipient of this e-mail, any disclosure, copying, distribution or use of its contents is strictly prohibited, and you should please notify the sender immediately and then delete it (including any attachments) from your system. Thank you.
> Am 01.06.2020 um 19:30 schrieb Paul Angus <pa...@shapeblue.com>:
>
> Hi Sven,
>
> I think that there is a piece of the jigsaw that you are missing.
>
> Given that the only thing that we know, is that we can no longer communicate with the host agent;  To avoid split brain/corruption of VMs, CloudStack must determine if the guests VMs are still running on the host not.  The only way we can do that is look for disk activity created by those VMs.
>
> Using a secondary storage heartbeat would give a false 'host is down' if say a switch went down carrying sec storage and mgmt. traffic
>
> Wrt solidfire, you could poll SolidFire via API for activity on the volumes which belong to the VMs on the unresponsive host.  I don't know if there is an equivalent for Ceph.
>
> Kind regards
>
>
> Paul Angus
>
>
>
> paul.angus@shapeblue.com
> www.shapeblue.com
> 3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK
> @shapeblue
>
>
>
>
> -----Original Message-----
> From: Sven Vogel <S....@ewerk.com>
> Sent: 01 June 2020 12:30
> To: dev <de...@cloudstack.apache.org>
> Subject: Managed Storage and HA
>
> Hi Community,
>
> I try to encounter how HA works. Our goal is it to make it usable with managed storage like (Netapp Solidfire / maybe it works with CEPH too) so if its possible.
>
> This is a good guide and for some times we fixed and added the missing keys.
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/High+Availability+Developer%27s+Guide<https://cwiki.apache.org/confluence/display/CLOUDSTACK/High+Availability+Developer's+Guide>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Host+HA
>
> In the database I found out there are three different types of HA.
>
> If you select in the configuration table "SELECT * FROM `configuration` WHERE `component` LIKE '%ha%' LIMIT 0,1000;“ you will get three types of components.
>
> 1. HAManager
> 2. HighAvailbilityManager
> 3. KVMHAConfig
>
> "HAManager and HighAvailbilityManager" are the base which was extended from Rohit with „KVMHAConfig“ - KVM with stonith fencing.
>
> I understand all things work together but maybe I need to understand the process a little bit better.
>
> ------------------------------------------------------------------------------------
> To clarify I write down what I think to each of them. This is what I understand but please correct me or help me to understand it a little bit better.
>
> —>I found out that if we use managed storage a restart of virtual machines only works on the same host. This is what I understand the lack of the missing heartbeat file on the shared storage because we don’t have shared storage like NFS.
>
> —
> "If the network ping investigation returns that it cannot detect the status of the host, CloudStack HA then relies on the hypervisor specific investigation. For VMware, there is no such investigation as the hypervisor host handles its own HA. For XenServer and KVM, CloudStack HA deploys a monitoring script that writes the current timestamp on to a heartbeat file on shared storage."
> —
>
> And
>
> —
> For the initial release, only KVM with NFS storage will be supported. However, the storage check component will be implemented in a modular fashion allowing for checks using other storage platforms(e.g. Ceph) in the future.
> —
> ------------------------------------------------------------------------------------
>
> We would implement a plugin or extend this for managed storage but at the moment I need to understand where this should happen. Since managed storage uses different volumes for each VM we should its not easy to make an storage heartbeat like NFS. the lack of one missing volume don’t means the hole storage has an problem so I think its not easy to encounter from one volumes to a complete storage.
>
> We don’t use KVMHAConfig a the moment and encounter that if a Host goes down (offline) the virtual machines will not be restarted on another host. They will only restarted on the host if the host comes back.(online). We don’t want a hard fencing of the hosts but we want a correct determination whether the host is still alive. Fencing would maybe in our case a little bit hard because we don’t have an hard data corruption on entire storage.
>
> Some questions.
> 1. let's assume correctly that the HA don’t work without an shared storage and network ping? Is this the cause why our virtual machines will not restarted on another host? Is this correct or do we have an config problem?
> 2. Where could the plugin be implemented? Is there a preferred place?
> 3. If point 1. Is correctly I thought the idea would be to add an global flag to use the secondary storage (NFS) as heartbeat to find out if there is any host inactive?
>
> Thanks and Cheers
>
> Sven
>
>
> __
>
> Sven Vogel
> Lead Cloud Solution Architect
>
> EWERK DIGITAL GmbH
> Brühl 24, D-04109 Leipzig
> P +49 341 42649 - 99
> F +49 341 42649 - 98
> S.Vogel@ewerk.com
> www.ewerk.com
>
> Geschäftsführer:
> Dr. Erik Wende, Hendrik Schubert, Tassilo Möschke
> Registergericht: Leipzig HRB 9065
>
> Support:
> +49 341 42649 555
>
> Zertifiziert nach:
> ISO/IEC 27001:2013
> DIN EN ISO 9001:2015
> DIN ISO/IEC 20000-1:2011
>
> ISAE 3402 Typ II Assessed
>
> EWERK-Blog<https://blog.ewerk.com/> | LinkedIn<https://www.linkedin.com/company/ewerk-group> | Xing<https://www.xing.com/company/ewerk> | Twitter<https://twitter.com/EWERK_Group> | Facebook<https://de-de.facebook.com/EWERK.IT/>
>
>
> Auskünfte und Angebote per Mail sind freibleibend und unverbindlich.
>
> Disclaimer Privacy:
> Der Inhalt dieser E-Mail (einschließlich etwaiger beigefügter Dateien) ist vertraulich und nur für den Empfänger bestimmt. Sollten Sie nicht der bestimmungsgemäße Empfänger sein, ist Ihnen jegliche Offenlegung, Vervielfältigung, Weitergabe oder Nutzung des Inhalts untersagt. Bitte informieren Sie in diesem Fall unverzüglich den Absender und löschen Sie die E-Mail (einschließlich etwaiger beigefügter Dateien) von Ihrem System. Vielen Dank.
>
> The contents of this e-mail (including any attachments) are confidential and may be legally privileged. If you are not the intended recipient of this e-mail, any disclosure, copying, distribution or use of its contents is strictly prohibited, and you should please notify the sender immediately and then delete it (including any attachments) from your system. Thank you.

RE: Managed Storage and HA

Posted by Paul Angus <pa...@shapeblue.com>.

Hi Sven,

I think that there is a piece of the jigsaw that you are missing.

Given that the only thing that we know, is that we can no longer communicate with the host agent;  To avoid split brain/corruption of VMs, CloudStack must determine if the guests VMs are still running on the host not.  The only way we can do that is look for disk activity created by those VMs.

Using a secondary storage heartbeat would give a false 'host is down' if say a switch went down carrying sec storage and mgmt. traffic

Wrt solidfire, you could poll SolidFire via API for activity on the volumes which belong to the VMs on the unresponsive host.  I don't know if there is an equivalent for Ceph.

Kind regards


Paul Angus



paul.angus@shapeblue.com 
www.shapeblue.com
3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK
@shapeblue
  
 


-----Original Message-----
From: Sven Vogel <S....@ewerk.com> 
Sent: 01 June 2020 12:30
To: dev <de...@cloudstack.apache.org>
Subject: Managed Storage and HA

Hi Community,

I try to encounter how HA works. Our goal is it to make it usable with managed storage like (Netapp Solidfire / maybe it works with CEPH too) so if its possible.

This is a good guide and for some times we fixed and added the missing keys.
https://cwiki.apache.org/confluence/display/CLOUDSTACK/High+Availability+Developer%27s+Guide<https://cwiki.apache.org/confluence/display/CLOUDSTACK/High+Availability+Developer's+Guide>
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Host+HA

In the database I found out there are three different types of HA.

If you select in the configuration table "SELECT * FROM `configuration` WHERE `component` LIKE '%ha%' LIMIT 0,1000;“ you will get three types of components.

1. HAManager
2. HighAvailbilityManager
3. KVMHAConfig

"HAManager and HighAvailbilityManager" are the base which was extended from Rohit with „KVMHAConfig“ - KVM with stonith fencing.

I understand all things work together but maybe I need to understand the process a little bit better.

------------------------------------------------------------------------------------
To clarify I write down what I think to each of them. This is what I understand but please correct me or help me to understand it a little bit better.

—>I found out that if we use managed storage a restart of virtual machines only works on the same host. This is what I understand the lack of the missing heartbeat file on the shared storage because we don’t have shared storage like NFS.

—
"If the network ping investigation returns that it cannot detect the status of the host, CloudStack HA then relies on the hypervisor specific investigation. For VMware, there is no such investigation as the hypervisor host handles its own HA. For XenServer and KVM, CloudStack HA deploys a monitoring script that writes the current timestamp on to a heartbeat file on shared storage."
—

And

—
For the initial release, only KVM with NFS storage will be supported. However, the storage check component will be implemented in a modular fashion allowing for checks using other storage platforms(e.g. Ceph) in the future.
—
------------------------------------------------------------------------------------

We would implement a plugin or extend this for managed storage but at the moment I need to understand where this should happen. Since managed storage uses different volumes for each VM we should its not easy to make an storage heartbeat like NFS. the lack of one missing volume don’t means the hole storage has an problem so I think its not easy to encounter from one volumes to a complete storage.

We don’t use KVMHAConfig a the moment and encounter that if a Host goes down (offline) the virtual machines will not be restarted on another host. They will only restarted on the host if the host comes back.(online). We don’t want a hard fencing of the hosts but we want a correct determination whether the host is still alive. Fencing would maybe in our case a little bit hard because we don’t have an hard data corruption on entire storage.

Some questions.
1. let's assume correctly that the HA don’t work without an shared storage and network ping? Is this the cause why our virtual machines will not restarted on another host? Is this correct or do we have an config problem?
2. Where could the plugin be implemented? Is there a preferred place?
3. If point 1. Is correctly I thought the idea would be to add an global flag to use the secondary storage (NFS) as heartbeat to find out if there is any host inactive?

Thanks and Cheers

Sven


__

Sven Vogel
Lead Cloud Solution Architect

EWERK DIGITAL GmbH
Brühl 24, D-04109 Leipzig
P +49 341 42649 - 99
F +49 341 42649 - 98
S.Vogel@ewerk.com
www.ewerk.com

Geschäftsführer:
Dr. Erik Wende, Hendrik Schubert, Tassilo Möschke
Registergericht: Leipzig HRB 9065

Support:
+49 341 42649 555

Zertifiziert nach:
ISO/IEC 27001:2013
DIN EN ISO 9001:2015
DIN ISO/IEC 20000-1:2011

ISAE 3402 Typ II Assessed

EWERK-Blog<https://blog.ewerk.com/> | LinkedIn<https://www.linkedin.com/company/ewerk-group> | Xing<https://www.xing.com/company/ewerk> | Twitter<https://twitter.com/EWERK_Group> | Facebook<https://de-de.facebook.com/EWERK.IT/>


Auskünfte und Angebote per Mail sind freibleibend und unverbindlich.

Disclaimer Privacy:
Der Inhalt dieser E-Mail (einschließlich etwaiger beigefügter Dateien) ist vertraulich und nur für den Empfänger bestimmt. Sollten Sie nicht der bestimmungsgemäße Empfänger sein, ist Ihnen jegliche Offenlegung, Vervielfältigung, Weitergabe oder Nutzung des Inhalts untersagt. Bitte informieren Sie in diesem Fall unverzüglich den Absender und löschen Sie die E-Mail (einschließlich etwaiger beigefügter Dateien) von Ihrem System. Vielen Dank.

The contents of this e-mail (including any attachments) are confidential and may be legally privileged. If you are not the intended recipient of this e-mail, any disclosure, copying, distribution or use of its contents is strictly prohibited, and you should please notify the sender immediately and then delete it (including any attachments) from your system. Thank you.