You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by "weizhouapache (via GitHub)" <gi...@apache.org> on 2026/03/09 12:38:18 UTC

[PR] KVM: Enable HA heartbeat on ShareMountPoint [cloudstack]

weizhouapache opened a new pull request, #12773:
URL: https://github.com/apache/cloudstack/pull/12773

   ### Description
   
   This PR improves KVM HA heartbeat to support SharedMountPoint
   
   
   Tested
   - add shared mount point pool
   - kvm hosts write heartbeat to the mount point
   - deploy vms
   - set global settingforce.ha to true
   - force shutdown one of the kvm hosts
   - after 3 mins, the VMs are started on other kvm hosts.
   
   Example of heartbeat files
   ```
   [root@kvm1 ~]# ls -l /mnt/cloudstack*/KVMHA
   /mnt/cloudstack1/KVMHA:
   total 1
   -rw-r--r--. 1 root root 11 Mar  9 12:28 hb-10.1.34.125
   -rw-r--r--. 1 root root 11 Mar  9 12:09 hb-10.1.35.76
   
   /mnt/cloudstack2/KVMHA:
   total 1
   -rw-r--r--. 1 root root 11 Mar  9 12:28 hb-10.1.34.125
   -rw-r--r--. 1 root root 11 Mar  9 12:09 hb-10.1.35.76
   ```
   <!--- Describe your changes in DETAIL - And how has behaviour functionally changed. -->
   
   <!-- For new features, provide link to FS, dev ML discussion etc. -->
   <!-- In case of bug fix, the expected and actual behaviours, steps to reproduce. -->
   
   <!-- When "Fixes: #<id>" is specified, the issue/PR will automatically be closed when this PR gets merged -->
   <!-- For addressing multiple issues/PRs, use multiple "Fixes: #<id>" -->
   <!-- Fixes: # -->
   
   <!--- ******************************************************************************* -->
   <!--- NOTE: AUTOMATION USES THE DESCRIPTIONS TO SET LABELS AND PRODUCE DOCUMENTATION. -->
   <!--- PLEASE PUT AN 'X' in only **ONE** box -->
   <!--- ******************************************************************************* -->
   
   ### Types of changes
   
   - [ ] Breaking change (fix or feature that would cause existing functionality to change)
   - [ ] New feature (non-breaking change which adds functionality)
   - [ ] Bug fix (non-breaking change which fixes an issue)
   - [x] Enhancement (improves an existing feature and functionality)
   - [ ] Cleanup (Code refactoring and cleanup, that may add test cases)
   - [ ] Build/CI
   - [ ] Test (unit or integration test code)
   
   ### Feature/Enhancement Scale or Bug Severity
   
   #### Feature/Enhancement Scale
   
   - [ ] Major
   - [ ] Minor
   
   #### Bug Severity
   
   - [ ] BLOCKER
   - [ ] Critical
   - [ ] Major
   - [ ] Minor
   - [ ] Trivial
   
   ### Screenshots (if appropriate):
   
   ### How Has This Been Tested?
   
   <!-- Please describe in detail how you tested your changes. -->
   <!-- Include details of your testing environment, and the tests you ran to -->
   
   #### How did you try to break this feature and the system with this change?
   
   <!-- see how your change affects other areas of the code, etc. -->
   
   <!-- Please read the [CONTRIBUTING](https://github.com/apache/cloudstack/blob/main/CONTRIBUTING.md) document -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KVM: Enable HA heartbeat on ShareMountPoint [cloudstack]

Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #12773:
URL: https://github.com/apache/cloudstack/pull/12773#issuecomment-4046986947

   @weizhouapache a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KVM: Enable HA heartbeat on ShareMountPoint [cloudstack]

Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on PR #12773:
URL: https://github.com/apache/cloudstack/pull/12773#issuecomment-4023951433

   
   @blueoranguran package
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KVM: Enable HA heartbeat on ShareMountPoint [cloudstack]

Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #12773:
URL: https://github.com/apache/cloudstack/pull/12773#issuecomment-4026195284

   Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 17054


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KVM: Enable HA heartbeat on ShareMountPoint [cloudstack]

Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on PR #12773:
URL: https://github.com/apache/cloudstack/pull/12773#issuecomment-4046325453

   @blueorangutan package


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KVM: Enable HA heartbeat on ShareMountPoint [cloudstack]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #12773:
URL: https://github.com/apache/cloudstack/pull/12773#discussion_r2906433893


##########
scripts/vm/hypervisor/kvm/kvmsmpheartbeat.sh:
##########
@@ -0,0 +1,218 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+help() {
+  printf "Usage: $0
+                -i identifier (ignored for local-only heartbeat)
+                -p path (ignored for local-only heartbeat)
+                -m mount point (local path where heartbeat will be written)
+                -h host (host IP/name to include in heartbeat filename)
+                -r write/read hb log (read-check mode)
+                -c cleanup (trigger emergency reboot)
+                -t interval between read hb log\n"
+  exit 1
+}
+
+#set -x
+NfsSvrIP=
+NfsSvrPath=
+MountPoint=
+HostIP=
+interval=
+rflag=0
+cflag=0
+
+while getopts 'i:p:m:h:t:rc' OPTION
+do
+  case $OPTION in
+  i)
+     NfsSvrIP="$OPTARG"
+     ;; # retained for CLI compatibility but unused for local-only script
+  p)
+     NfsSvrPath="$OPTARG"
+     ;; # retained for CLI compatibility but unused for local-only script
+  m)
+     MountPoint="$OPTARG"
+     ;;
+  h)
+     HostIP="$OPTARG"
+     ;;
+  r)
+     rflag=1
+     ;;
+  t)
+     interval="$OPTARG"
+     ;;
+  c)
+    cflag=1
+     ;;
+  *)
+     help
+     ;;
+  esac
+done
+
+# Match original kvmheartbeat.sh: require NfsSvrIP parameter for CLI compatibility
+if [ -z "$NfsSvrIP" ]
+then
+   exit 1
+fi
+
+# For local-only heartbeat we require a mountpoint
+if [ -z "$MountPoint" ]
+then
+   echo "Mount point (-m) is required"
+   help
+fi
+
+# Ensure mount point exists and is writable
+if [ ! -d "$MountPoint" ]; then
+  mkdir -p "$MountPoint" 2>/dev/null
+  if [ $? -ne 0 ]; then
+    echo "Failed to create mount point directory: $MountPoint" >&2
+    exit 1
+  fi
+fi
+
+# Determine a sensible HostIP if not provided
+if [ -z "$HostIP" ]; then
+  # try to get a non-loopback IPv4 address, fallback to hostname
+  ipaddr=$(hostname -I 2>/dev/null | awk '{print $1}')
+  if [ -n "$ipaddr" ]; then
+    HostIP="$ipaddr"
+  else
+    HostIP=$(hostname)
+  fi
+fi
+
+#delete VMs on this mountpoint (best-effort)
+deleteVMs() {
+  local mountPoint=$1
+  vmPids=$(ps aux | grep qemu | grep "$mountPoint" | awk '{print $2}' 2> /dev/null)
+  if [ $? -gt 0 ]
+  then
+     return
+  fi
+
+  if [ -z "$vmPids" ]
+  then
+     return
+  fi
+
+  for pid in $vmPids
+  do
+     kill -9 $pid &> /dev/null
+  done
+}
+
+#checking is there the mount point present under $MountPoint?
+mounts=$(cat /proc/mounts | grep "$MountPoint")
+if [ $? -gt 0 ]
+then
+   # mount point not present — we don't remount in local-only script
+   # nothing to do here; keep for compatibility with original flow
+   :
+else
+   # mount exists; if not in read-check mode, consider deleting VMs similar to original behavior
+   if [ "$rflag" == "0" ]
+   then
+     deleteVMs $MountPoint
+   fi
+fi
+
+hbFolder="$MountPoint/KVMHA/"
+hbFile="$hbFolder/hb-$HostIP"
+
+write_hbLog() {
+#write the heart beat log
+  stat "$hbFile" &> /dev/null
+  if [ $? -gt 0 ]
+  then
+     # create a new one
+     mkdir -p "$hbFolder" &> /dev/null
+     # touch will be done by atomic write below; ensure folder is writable
+     if [ ! -w "$hbFolder" ]; then
+       printf "Folder not writable: $hbFolder" >&2
+       return 2
+     fi
+  fi
+
+  timestamp=$(date +%s)
+  # Write atomically to avoid partial writes (write to tmp then mv)
+  tmpfile="${hbFile}.$$"
+  printf "%s\n" "$timestamp" > "$tmpfile" 2>/dev/null
+  if [ $? -ne 0 ]; then
+    printf "Failed to write heartbeat to $tmpfile" >&2
+    return 2
+  fi
+  mv -f "$tmpfile" "$hbFile" 2>/dev/null
+  return $?
+}
+
+check_hbLog() {
+  if [ ! -f "$hbFile" ]; then
+    # signal large difference if file missing
+    return 999999
+  fi
+  now=$(date +%s)
+  hb=$(cat "$hbFile" 2>/dev/null)
+  if [ -z "$hb" ]; then
+    return 999998
+  fi
+  diff=`expr $now - $hb 2>/dev/null`
+  if [ $? -ne 0 ]
+  then
+    return 999997
+  fi
+  if [ -z "$interval" ]; then
+    # if no interval provided, consider 0 as success
+    if [ $diff -gt 0 ]; then
+      return $diff
+    else
+      return 0
+    fi
+  fi
+  if [ $diff -gt $interval ]
+  then
+    return $diff
+  fi
+  return 0
+}
+
+if [ "$rflag" == "1" ]
+then
+  check_hbLog
+  diff=$?
+  if [ $diff == 0 ]

Review Comment:
   `return` in bash only supports exit codes 0–255; values like 999999 will wrap modulo 256, so `diff=$?` will not contain the intended “seconds ago” value and can lead to incorrect DEAD/ALIVE reporting. Consider having `check_hbLog` print the computed diff (e.g., via `echo`) and use its return code only as success/failure, or store the diff in a global variable and return 0/1.
   ```suggestion
     hb_diff=0
     if [ ! -f "$hbFile" ]; then
       # signal large difference if file missing
       hb_diff=999999
       return 1
     fi
     now=$(date +%s)
     hb=$(cat "$hbFile" 2>/dev/null)
     if [ -z "$hb" ]; then
       hb_diff=999998
       return 1
     fi
     diff=`expr $now - $hb 2>/dev/null`
     if [ $? -ne 0 ]
     then
       hb_diff=999997
       return 1
     fi
     if [ -z "$interval" ]; then
       # if no interval provided, consider 0 as success
       if [ $diff -gt 0 ]; then
         hb_diff=$diff
         return 1
       else
         hb_diff=0
         return 0
       fi
     fi
     if [ $diff -gt $interval ]
     then
       hb_diff=$diff
       return 1
     fi
     hb_diff=0
     return 0
   }
   
   if [ "$rflag" == "1" ]
   then
     check_hbLog
     status=$?
     diff="${hb_diff:-0}"
     if [ $status -eq 0 ]
   ```



##########
scripts/vm/hypervisor/kvm/kvmsmpheartbeat.sh:
##########
@@ -0,0 +1,218 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+help() {
+  printf "Usage: $0
+                -i identifier (ignored for local-only heartbeat)
+                -p path (ignored for local-only heartbeat)
+                -m mount point (local path where heartbeat will be written)
+                -h host (host IP/name to include in heartbeat filename)
+                -r write/read hb log (read-check mode)
+                -c cleanup (trigger emergency reboot)
+                -t interval between read hb log\n"
+  exit 1
+}
+
+#set -x
+NfsSvrIP=
+NfsSvrPath=
+MountPoint=
+HostIP=
+interval=
+rflag=0
+cflag=0
+
+while getopts 'i:p:m:h:t:rc' OPTION
+do
+  case $OPTION in
+  i)
+     NfsSvrIP="$OPTARG"
+     ;; # retained for CLI compatibility but unused for local-only script
+  p)
+     NfsSvrPath="$OPTARG"
+     ;; # retained for CLI compatibility but unused for local-only script
+  m)
+     MountPoint="$OPTARG"
+     ;;
+  h)
+     HostIP="$OPTARG"
+     ;;
+  r)
+     rflag=1
+     ;;
+  t)
+     interval="$OPTARG"
+     ;;
+  c)
+    cflag=1
+     ;;
+  *)
+     help
+     ;;
+  esac
+done
+
+# Match original kvmheartbeat.sh: require NfsSvrIP parameter for CLI compatibility
+if [ -z "$NfsSvrIP" ]
+then
+   exit 1
+fi
+
+# For local-only heartbeat we require a mountpoint
+if [ -z "$MountPoint" ]
+then
+   echo "Mount point (-m) is required"
+   help
+fi
+
+# Ensure mount point exists and is writable
+if [ ! -d "$MountPoint" ]; then
+  mkdir -p "$MountPoint" 2>/dev/null
+  if [ $? -ne 0 ]; then
+    echo "Failed to create mount point directory: $MountPoint" >&2
+    exit 1
+  fi
+fi
+
+# Determine a sensible HostIP if not provided
+if [ -z "$HostIP" ]; then
+  # try to get a non-loopback IPv4 address, fallback to hostname
+  ipaddr=$(hostname -I 2>/dev/null | awk '{print $1}')
+  if [ -n "$ipaddr" ]; then
+    HostIP="$ipaddr"
+  else
+    HostIP=$(hostname)
+  fi
+fi
+
+#delete VMs on this mountpoint (best-effort)
+deleteVMs() {
+  local mountPoint=$1
+  vmPids=$(ps aux | grep qemu | grep "$mountPoint" | awk '{print $2}' 2> /dev/null)
+  if [ $? -gt 0 ]
+  then
+     return
+  fi
+
+  if [ -z "$vmPids" ]
+  then
+     return
+  fi
+
+  for pid in $vmPids
+  do
+     kill -9 $pid &> /dev/null
+  done
+}
+
+#checking is there the mount point present under $MountPoint?
+mounts=$(cat /proc/mounts | grep "$MountPoint")
+if [ $? -gt 0 ]
+then
+   # mount point not present — we don't remount in local-only script
+   # nothing to do here; keep for compatibility with original flow
+   :
+else

Review Comment:
   `grep "$MountPoint"` can produce false positives when `$MountPoint` is a substring of another mount target (e.g., `/mnt/cloudstack1` matching `/mnt/cloudstack10`). This can incorrectly treat the mount as present and trigger `deleteVMs`. Consider using a stricter mount check (e.g., matching the mount target field exactly or using `findmnt`).



##########
scripts/vm/hypervisor/kvm/kvmsmpheartbeat.sh:
##########
@@ -0,0 +1,218 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+help() {
+  printf "Usage: $0
+                -i identifier (ignored for local-only heartbeat)
+                -p path (ignored for local-only heartbeat)
+                -m mount point (local path where heartbeat will be written)
+                -h host (host IP/name to include in heartbeat filename)
+                -r write/read hb log (read-check mode)
+                -c cleanup (trigger emergency reboot)
+                -t interval between read hb log\n"
+  exit 1
+}
+
+#set -x
+NfsSvrIP=
+NfsSvrPath=
+MountPoint=
+HostIP=
+interval=
+rflag=0
+cflag=0
+
+while getopts 'i:p:m:h:t:rc' OPTION
+do
+  case $OPTION in
+  i)
+     NfsSvrIP="$OPTARG"
+     ;; # retained for CLI compatibility but unused for local-only script
+  p)
+     NfsSvrPath="$OPTARG"
+     ;; # retained for CLI compatibility but unused for local-only script
+  m)
+     MountPoint="$OPTARG"
+     ;;
+  h)
+     HostIP="$OPTARG"
+     ;;
+  r)
+     rflag=1
+     ;;
+  t)
+     interval="$OPTARG"
+     ;;
+  c)
+    cflag=1
+     ;;
+  *)
+     help
+     ;;
+  esac
+done
+
+# Match original kvmheartbeat.sh: require NfsSvrIP parameter for CLI compatibility
+if [ -z "$NfsSvrIP" ]
+then
+   exit 1
+fi
+

Review Comment:
   The help text says `-i` is “ignored for local-only heartbeat”, but the script hard-requires it and exits with status 1 without any message. Consider either (a) not requiring `-i` for this script, or (b) emitting a clear error (and/or calling `help`) explaining why it’s required for compatibility.
   ```suggestion
   
   ```



##########
plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/KVMStoragePoolManager.java:
##########
@@ -57,7 +57,7 @@ public class KVMStoragePoolManager {
     private final Map<String, StoragePoolInformation> _storagePools = new ConcurrentHashMap<String, StoragePoolInformation>();
     private final Map<String, StorageAdaptor> _storageMapper = new HashMap<String, StorageAdaptor>();
 
-    private StorageAdaptor getStorageAdaptor(StoragePoolType type) {
+    public StorageAdaptor getStorageAdaptor(StoragePoolType type) {

Review Comment:
   Changing this method from `private` to `public` expands the exposed API surface of `KVMStoragePoolManager`, which can make future refactors harder. If external access is required, consider narrowing visibility (package-private) or providing a more purpose-driven public method; if it’s only needed for tests, consider a test-oriented visibility approach rather than making it broadly public.
   ```suggestion
       StorageAdaptor getStorageAdaptor(StoragePoolType type) {
   ```



##########
scripts/vm/hypervisor/kvm/kvmsmpheartbeat.sh:
##########
@@ -0,0 +1,218 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+help() {
+  printf "Usage: $0
+                -i identifier (ignored for local-only heartbeat)
+                -p path (ignored for local-only heartbeat)
+                -m mount point (local path where heartbeat will be written)
+                -h host (host IP/name to include in heartbeat filename)
+                -r write/read hb log (read-check mode)
+                -c cleanup (trigger emergency reboot)
+                -t interval between read hb log\n"
+  exit 1
+}
+
+#set -x
+NfsSvrIP=
+NfsSvrPath=
+MountPoint=
+HostIP=
+interval=
+rflag=0
+cflag=0
+
+while getopts 'i:p:m:h:t:rc' OPTION
+do
+  case $OPTION in
+  i)
+     NfsSvrIP="$OPTARG"
+     ;; # retained for CLI compatibility but unused for local-only script
+  p)
+     NfsSvrPath="$OPTARG"
+     ;; # retained for CLI compatibility but unused for local-only script
+  m)
+     MountPoint="$OPTARG"
+     ;;
+  h)
+     HostIP="$OPTARG"
+     ;;
+  r)
+     rflag=1
+     ;;
+  t)
+     interval="$OPTARG"
+     ;;
+  c)
+    cflag=1
+     ;;
+  *)
+     help
+     ;;
+  esac
+done
+
+# Match original kvmheartbeat.sh: require NfsSvrIP parameter for CLI compatibility
+if [ -z "$NfsSvrIP" ]
+then
+   exit 1
+fi
+
+# For local-only heartbeat we require a mountpoint
+if [ -z "$MountPoint" ]
+then
+   echo "Mount point (-m) is required"
+   help
+fi
+
+# Ensure mount point exists and is writable
+if [ ! -d "$MountPoint" ]; then
+  mkdir -p "$MountPoint" 2>/dev/null
+  if [ $? -ne 0 ]; then
+    echo "Failed to create mount point directory: $MountPoint" >&2
+    exit 1
+  fi
+fi
+
+# Determine a sensible HostIP if not provided
+if [ -z "$HostIP" ]; then
+  # try to get a non-loopback IPv4 address, fallback to hostname
+  ipaddr=$(hostname -I 2>/dev/null | awk '{print $1}')
+  if [ -n "$ipaddr" ]; then
+    HostIP="$ipaddr"
+  else
+    HostIP=$(hostname)
+  fi
+fi
+
+#delete VMs on this mountpoint (best-effort)
+deleteVMs() {
+  local mountPoint=$1
+  vmPids=$(ps aux | grep qemu | grep "$mountPoint" | awk '{print $2}' 2> /dev/null)
+  if [ $? -gt 0 ]
+  then
+     return
+  fi
+
+  if [ -z "$vmPids" ]
+  then
+     return
+  fi
+
+  for pid in $vmPids
+  do
+     kill -9 $pid &> /dev/null
+  done
+}
+
+#checking is there the mount point present under $MountPoint?
+mounts=$(cat /proc/mounts | grep "$MountPoint")
+if [ $? -gt 0 ]
+then
+   # mount point not present — we don't remount in local-only script
+   # nothing to do here; keep for compatibility with original flow
+   :
+else
+   # mount exists; if not in read-check mode, consider deleting VMs similar to original behavior
+   if [ "$rflag" == "0" ]
+   then
+     deleteVMs $MountPoint
+   fi
+fi
+
+hbFolder="$MountPoint/KVMHA/"
+hbFile="$hbFolder/hb-$HostIP"
+
+write_hbLog() {
+#write the heart beat log
+  stat "$hbFile" &> /dev/null
+  if [ $? -gt 0 ]
+  then
+     # create a new one
+     mkdir -p "$hbFolder" &> /dev/null
+     # touch will be done by atomic write below; ensure folder is writable
+     if [ ! -w "$hbFolder" ]; then
+       printf "Folder not writable: $hbFolder" >&2
+       return 2
+     fi
+  fi
+
+  timestamp=$(date +%s)
+  # Write atomically to avoid partial writes (write to tmp then mv)
+  tmpfile="${hbFile}.$$"
+  printf "%s\n" "$timestamp" > "$tmpfile" 2>/dev/null
+  if [ $? -ne 0 ]; then
+    printf "Failed to write heartbeat to $tmpfile" >&2
+    return 2
+  fi
+  mv -f "$tmpfile" "$hbFile" 2>/dev/null
+  return $?
+}
+
+check_hbLog() {
+  if [ ! -f "$hbFile" ]; then
+    # signal large difference if file missing
+    return 999999
+  fi
+  now=$(date +%s)
+  hb=$(cat "$hbFile" 2>/dev/null)
+  if [ -z "$hb" ]; then
+    return 999998
+  fi
+  diff=`expr $now - $hb 2>/dev/null`
+  if [ $? -ne 0 ]
+  then
+    return 999997
+  fi
+  if [ -z "$interval" ]; then
+    # if no interval provided, consider 0 as success
+    if [ $diff -gt 0 ]; then
+      return $diff
+    else
+      return 0
+    fi
+  fi
+  if [ $diff -gt $interval ]
+  then
+    return $diff
+  fi
+  return 0
+}
+
+if [ "$rflag" == "1" ]
+then
+  check_hbLog
+  diff=$?
+  if [ $diff == 0 ]
+  then
+    echo "=====> ALIVE <====="
+  else
+    echo "=====> Considering host as DEAD because last write on [$hbFile] was [$diff] seconds ago, but the max interval is [$interval] <======"
+  fi

Review Comment:
   This is doing a numeric comparison but uses the string operator `==` and unquoted `$diff`. Use numeric operators (`-eq`, `-ne`) and quote variables in `[` tests (or switch to `[[ ... ]]`) to avoid mis-comparisons or test errors if the variable is empty/unset.



##########
scripts/vm/hypervisor/kvm/kvmsmpheartbeat.sh:
##########
@@ -0,0 +1,218 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+help() {
+  printf "Usage: $0
+                -i identifier (ignored for local-only heartbeat)
+                -p path (ignored for local-only heartbeat)

Review Comment:
   The help text says `-i` is “ignored for local-only heartbeat”, but the script hard-requires it and exits with status 1 without any message. Consider either (a) not requiring `-i` for this script, or (b) emitting a clear error (and/or calling `help`) explaining why it’s required for compatibility.
   ```suggestion
                   -i identifier (required for CLI compatibility; value ignored by local-only heartbeat)
                   -p path (required for CLI compatibility; value ignored by local-only heartbeat)
   ```



##########
plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/KVMHAMonitor.java:
##########
@@ -34,6 +34,8 @@
 
 public class KVMHAMonitor extends KVMHABase implements Runnable {
 
+    public static final List<StoragePoolType> STORAGE_POOL_TYPES_WITH_HA_SUPPORT = List.of(StoragePoolType.NetworkFilesystem, StoragePoolType.SharedMountPoint);

Review Comment:
   HA-supported pool types are now defined in multiple places (e.g., also in `CloudStackPrimaryDataStoreDriverImpl`), and `LibvirtStoragePool` depends on `KVMHAMonitor` just to reuse this constant. Consider centralizing this list in a small shared utility/helper in an appropriate common package for the KVM plugin (or a single authoritative location) to avoid duplication and reduce cross-class coupling.



##########
scripts/vm/hypervisor/kvm/kvmsmpheartbeat.sh:
##########
@@ -0,0 +1,218 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+help() {
+  printf "Usage: $0
+                -i identifier (ignored for local-only heartbeat)
+                -p path (ignored for local-only heartbeat)
+                -m mount point (local path where heartbeat will be written)
+                -h host (host IP/name to include in heartbeat filename)
+                -r write/read hb log (read-check mode)
+                -c cleanup (trigger emergency reboot)
+                -t interval between read hb log\n"
+  exit 1
+}
+
+#set -x
+NfsSvrIP=
+NfsSvrPath=
+MountPoint=
+HostIP=
+interval=
+rflag=0
+cflag=0
+
+while getopts 'i:p:m:h:t:rc' OPTION
+do
+  case $OPTION in
+  i)
+     NfsSvrIP="$OPTARG"
+     ;; # retained for CLI compatibility but unused for local-only script
+  p)
+     NfsSvrPath="$OPTARG"
+     ;; # retained for CLI compatibility but unused for local-only script
+  m)
+     MountPoint="$OPTARG"
+     ;;
+  h)
+     HostIP="$OPTARG"
+     ;;
+  r)
+     rflag=1
+     ;;
+  t)
+     interval="$OPTARG"
+     ;;
+  c)
+    cflag=1
+     ;;
+  *)
+     help
+     ;;
+  esac
+done
+
+# Match original kvmheartbeat.sh: require NfsSvrIP parameter for CLI compatibility
+if [ -z "$NfsSvrIP" ]
+then
+   exit 1
+fi
+
+# For local-only heartbeat we require a mountpoint
+if [ -z "$MountPoint" ]
+then
+   echo "Mount point (-m) is required"
+   help
+fi
+
+# Ensure mount point exists and is writable
+if [ ! -d "$MountPoint" ]; then
+  mkdir -p "$MountPoint" 2>/dev/null
+  if [ $? -ne 0 ]; then
+    echo "Failed to create mount point directory: $MountPoint" >&2
+    exit 1
+  fi
+fi
+
+# Determine a sensible HostIP if not provided
+if [ -z "$HostIP" ]; then
+  # try to get a non-loopback IPv4 address, fallback to hostname
+  ipaddr=$(hostname -I 2>/dev/null | awk '{print $1}')
+  if [ -n "$ipaddr" ]; then
+    HostIP="$ipaddr"
+  else
+    HostIP=$(hostname)
+  fi
+fi
+
+#delete VMs on this mountpoint (best-effort)
+deleteVMs() {
+  local mountPoint=$1
+  vmPids=$(ps aux | grep qemu | grep "$mountPoint" | awk '{print $2}' 2> /dev/null)
+  if [ $? -gt 0 ]
+  then
+     return
+  fi

Review Comment:
   `$?` here reflects only the exit status of the last command in the pipeline (`awk`), not whether `ps`/`grep` matched anything. As written, this check won’t behave as intended and is effectively redundant with the later `-z "$vmPids"` guard. Consider removing this `$?` block, or use `set -o pipefail` (with care for script-wide impact) or replace the pipeline with a more direct PID lookup (e.g., `pgrep`) and check for empty output.
   ```suggestion
   
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KVM: Enable HA heartbeat on ShareMountPoint [cloudstack]

Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #12773:
URL: https://github.com/apache/cloudstack/pull/12773#issuecomment-4137025926

   Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 17254


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KVM: Enable HA heartbeat on ShareMountPoint [cloudstack]

Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #12773:
URL: https://github.com/apache/cloudstack/pull/12773#issuecomment-4026862671

   @weizhouapache a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KVM: Enable HA heartbeat on ShareMountPoint [cloudstack]

Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland commented on code in PR #12773:
URL: https://github.com/apache/cloudstack/pull/12773#discussion_r2912260918


##########
plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/KVMStoragePoolManager.java:
##########
@@ -57,7 +57,7 @@ public class KVMStoragePoolManager {
     private final Map<String, StoragePoolInformation> _storagePools = new ConcurrentHashMap<String, StoragePoolInformation>();
     private final Map<String, StorageAdaptor> _storageMapper = new HashMap<String, StorageAdaptor>();
 
-    private StorageAdaptor getStorageAdaptor(StoragePoolType type) {
+    public StorageAdaptor getStorageAdaptor(StoragePoolType type) {

Review Comment:
   have to agree with the slop here. I don’t see this used outside this file.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KVM: Enable HA heartbeat on ShareMountPoint [cloudstack]

Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on PR #12773:
URL: https://github.com/apache/cloudstack/pull/12773#issuecomment-4046983640

   @blueorangutan test


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KVM: Enable HA heartbeat on ShareMountPoint [cloudstack]

Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on PR #12773:
URL: https://github.com/apache/cloudstack/pull/12773#issuecomment-4025828763

   @blueorangutan package


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KVM: Enable HA heartbeat on ShareMountPoint [cloudstack]

Posted by "sureshanaparti (via GitHub)" <gi...@apache.org>.
sureshanaparti commented on code in PR #12773:
URL: https://github.com/apache/cloudstack/pull/12773#discussion_r2922982736


##########
plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/KVMHAMonitor.java:
##########
@@ -34,6 +34,8 @@
 
 public class KVMHAMonitor extends KVMHABase implements Runnable {
 
+    public static final List<StoragePoolType> STORAGE_POOL_TYPES_WITH_HA_SUPPORT = List.of(StoragePoolType.NetworkFilesystem, StoragePoolType.SharedMountPoint);

Review Comment:
   @weizhouapache is it possible to add supportsHA() in respective storage driver/plugin and check for it? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KVM: Enable HA heartbeat on ShareMountPoint [cloudstack]

Posted by "sureshanaparti (via GitHub)" <gi...@apache.org>.
sureshanaparti commented on code in PR #12773:
URL: https://github.com/apache/cloudstack/pull/12773#discussion_r2922982736


##########
plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/KVMHAMonitor.java:
##########
@@ -34,6 +34,8 @@
 
 public class KVMHAMonitor extends KVMHABase implements Runnable {
 
+    public static final List<StoragePoolType> STORAGE_POOL_TYPES_WITH_HA_SUPPORT = List.of(StoragePoolType.NetworkFilesystem, StoragePoolType.SharedMountPoint);

Review Comment:
   @weizhouapache is it possible to add supportHA() in respective storage driver/plugin and check for it? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KVM: Enable HA heartbeat on ShareMountPoint [cloudstack]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #12773:
URL: https://github.com/apache/cloudstack/pull/12773#discussion_r2923378227


##########
plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/KVMStoragePool.java:
##########
@@ -91,6 +91,9 @@ default Long getUsedIops() {
 
     public StoragePoolType getType();
 
+    default void setType(StoragePoolType type) {

Review Comment:
   `setType` is introduced as a default no-op method. Call sites (e.g., `pool.setType(type)`) appear to rely on it to actually mutate the pool, but any `KVMStoragePool` implementation that forgets to override it will silently ignore the update. Consider making this an abstract method (and implementing it across pool types) or at least documenting clearly that it must be overridden when the implementation stores `type` internally.
   ```suggestion
   
       /**
        * Set the type of this storage pool.
        * <p>
        * Implementations that store the pool type internally must override this
        * method to ensure {@link #getType()} reflects the updated value.
        * The default implementation throws {@link UnsupportedOperationException}
        * to avoid silently ignoring type updates.
        *
        * @param type the {@link StoragePoolType} to set for this pool
        */
       default void setType(StoragePoolType type) {
           throw new UnsupportedOperationException("setType(StoragePoolType) must be overridden by KVMStoragePool implementations that support changing the pool type");
   ```



##########
scripts/vm/hypervisor/kvm/kvmsmpheartbeat.sh:
##########
@@ -0,0 +1,205 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+help() {
+  printf "Usage: $0
+                -i identifier (required for CLI compatibility; value ignored by local-only heartbeat)
+                -p path (required for CLI compatibility; value ignored by local-only heartbeat)
+                -m mount point (local path where heartbeat will be written)
+                -h host (host IP/name to include in heartbeat filename)
+                -r write/read hb log (read-check mode)
+                -c cleanup (trigger emergency reboot)
+                -t interval between read hb log\n"
+  exit 1
+}
+
+#set -x
+NfsSvrIP=
+NfsSvrPath=
+MountPoint=
+HostIP=
+interval=
+rflag=0
+cflag=0
+
+while getopts 'i:p:m:h:t:rc' OPTION
+do
+  case $OPTION in
+  i)
+     NfsSvrIP="$OPTARG"
+     ;; # retained for CLI compatibility but unused for this script
+  p)
+     NfsSvrPath="$OPTARG"
+     ;; # retained for CLI compatibility but unused for this script
+  m)
+     MountPoint="$OPTARG"
+     ;;
+  h)
+     HostIP="$OPTARG"
+     ;;
+  r)
+     rflag=1
+     ;;
+  t)
+     interval="$OPTARG"
+     ;;
+  c)
+    cflag=1
+     ;;
+  *)
+     help
+     ;;
+  esac
+done
+
+# For heartbeat we require a mountpoint
+if [ -z "$MountPoint" ]
+then
+   echo "Mount point (-m) is required"
+   help
+fi
+
+# Ensure mount point exists and is writable
+if [ ! -d "$MountPoint" ]; then
+  mkdir -p "$MountPoint" 2>/dev/null
+  if [ $? -ne 0 ]; then
+    echo "Failed to create mount point directory: $MountPoint" >&2
+    exit 1
+  fi
+fi
+
+#delete VMs on this mountpoint (best-effort)
+deleteVMs() {
+  local mountPoint=$1
+  vmPids=$(ps aux | grep qemu | grep "$mountPoint" | awk '{print $2}' 2> /dev/null)
+
+  if [ -z "$vmPids" ]
+  then
+     return
+  fi
+
+  for pid in $vmPids
+  do
+     kill -9 $pid &> /dev/null
+  done
+}
+
+#checking is there the mount point present under $MountPoint?
+if grep -q "^[^ ]\+ $MountPoint " /proc/mounts
+then
+   # mount exists; if not in read-check mode, consider deleting VMs similar to original behavior
+   if [ "$rflag" == "0" ]
+   then
+     deleteVMs $MountPoint
+   fi
+else
+   # mount point not present — we don't remount in local-only script
+   # nothing to do here; keep for compatibility with original flow
+   :
+fi
+
+hbFolder="$MountPoint/KVMHA/"
+hbFile="$hbFolder/hb-$HostIP"
+
+write_hbLog() {
+#write the heart beat log
+  stat "$hbFile" &> /dev/null
+  if [ $? -gt 0 ]
+  then
+     # create a new one
+     mkdir -p "$hbFolder" &> /dev/null
+     # touch will be done by atomic write below; ensure folder is writable
+     if [ ! -w "$hbFolder" ]; then
+       printf "Folder not writable: $hbFolder" >&2
+       return 2
+     fi
+  fi
+
+  timestamp=$(date +%s)
+  # Write atomically to avoid partial writes (write to tmp then mv)
+  tmpfile="${hbFile}.$$"
+  printf "%s\n" "$timestamp" > "$tmpfile" 2>/dev/null
+  if [ $? -ne 0 ]; then
+    printf "Failed to write heartbeat to $tmpfile" >&2
+    return 2
+  fi
+  mv -f "$tmpfile" "$hbFile" 2>/dev/null
+  return $?
+}
+
+check_hbLog() {
+  hb_diff=0
+  if [ ! -f "$hbFile" ]; then
+    # signal large difference if file missing
+    hb_diff=999999
+    return 1
+  fi
+  now=$(date +%s)
+  hb=$(cat "$hbFile" 2>/dev/null)
+  if [ -z "$hb" ]; then
+    hb_diff=999998
+    return 1
+  fi
+  diff=`expr $now - $hb 2>/dev/null`
+  if [ $? -ne 0 ]
+  then
+    hb_diff=999997
+    return 1
+  fi
+  if [ -z "$interval" ]; then
+    # if no interval provided, consider 0 as success
+    if [ $diff -gt 0 ]; then
+      hb_diff=$diff
+      return 1
+    else
+      hb_diff=0
+      return 0
+    fi

Review Comment:
   In read-check mode, `interval` is expected to be provided (`-t`), but if it is empty this logic will return DEAD for almost any normal `diff > 0` (which will be true immediately after the first second). Consider making `-t` mandatory when `-r` is used (fail fast with usage/help), or define a sensible default interval and use it consistently.
   ```suggestion
       # if no interval provided, use a sensible default interval (in seconds)
       interval=60
   ```



##########
plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/LibvirtStoragePool.java:
##########
@@ -320,13 +321,16 @@ public void setDetails(Map<String, String> details) {
 
     @Override
     public boolean isPoolSupportHA() {
-        return type == StoragePoolType.NetworkFilesystem;
+        return HighAvailabilityManager.LIBVIRT_STORAGE_POOL_TYPES_WITH_HA_SUPPORT.contains(type);
     }
 
     public String getHearthBeatPath() {
-        if (type == StoragePoolType.NetworkFilesystem) {
+        if (StoragePoolType.NetworkFilesystem.equals(type)) {
             String kvmScriptsDir = AgentPropertiesFileHandler.getPropertyValue(AgentProperties.KVM_SCRIPTS_DIR);
             return Script.findScript(kvmScriptsDir, "kvmheartbeat.sh");
+        } else if (StoragePoolType.SharedMountPoint.equals(type)) {
+            String kvmScriptsDir = AgentPropertiesFileHandler.getPropertyValue(AgentProperties.KVM_SCRIPTS_DIR);
+            return Script.findScript(kvmScriptsDir, "kvmsmpheartbeat.sh");

Review Comment:
   `Script.findScript(...)` can return `null` if the heartbeat script is missing. Returning `null` here will later result in constructing a `Script` with a null command, which can fail in a less obvious way at runtime. Consider checking for a null script path and failing fast with a clear exception/error so mispackaging/misconfiguration is detected early.
   ```suggestion
               String scriptPath = Script.findScript(kvmScriptsDir, "kvmheartbeat.sh");
               if (scriptPath == null) {
                   throw new CloudRuntimeException("Unable to find heartbeat script 'kvmheartbeat.sh' in directory: " + kvmScriptsDir);
               }
               return scriptPath;
           } else if (StoragePoolType.SharedMountPoint.equals(type)) {
               String kvmScriptsDir = AgentPropertiesFileHandler.getPropertyValue(AgentProperties.KVM_SCRIPTS_DIR);
               String scriptPath = Script.findScript(kvmScriptsDir, "kvmsmpheartbeat.sh");
               if (scriptPath == null) {
                   throw new CloudRuntimeException("Unable to find heartbeat script 'kvmsmpheartbeat.sh' in directory: " + kvmScriptsDir);
               }
               return scriptPath;
   ```



##########
scripts/vm/hypervisor/kvm/kvmsmpheartbeat.sh:
##########
@@ -0,0 +1,205 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+help() {
+  printf "Usage: $0
+                -i identifier (required for CLI compatibility; value ignored by local-only heartbeat)
+                -p path (required for CLI compatibility; value ignored by local-only heartbeat)
+                -m mount point (local path where heartbeat will be written)
+                -h host (host IP/name to include in heartbeat filename)
+                -r write/read hb log (read-check mode)
+                -c cleanup (trigger emergency reboot)
+                -t interval between read hb log\n"
+  exit 1
+}
+
+#set -x
+NfsSvrIP=
+NfsSvrPath=
+MountPoint=
+HostIP=
+interval=
+rflag=0
+cflag=0
+
+while getopts 'i:p:m:h:t:rc' OPTION
+do
+  case $OPTION in
+  i)
+     NfsSvrIP="$OPTARG"
+     ;; # retained for CLI compatibility but unused for this script
+  p)
+     NfsSvrPath="$OPTARG"
+     ;; # retained for CLI compatibility but unused for this script
+  m)
+     MountPoint="$OPTARG"
+     ;;
+  h)
+     HostIP="$OPTARG"
+     ;;
+  r)
+     rflag=1
+     ;;
+  t)
+     interval="$OPTARG"
+     ;;
+  c)
+    cflag=1
+     ;;
+  *)
+     help
+     ;;
+  esac
+done
+
+# For heartbeat we require a mountpoint
+if [ -z "$MountPoint" ]
+then
+   echo "Mount point (-m) is required"
+   help
+fi
+
+# Ensure mount point exists and is writable
+if [ ! -d "$MountPoint" ]; then
+  mkdir -p "$MountPoint" 2>/dev/null
+  if [ $? -ne 0 ]; then
+    echo "Failed to create mount point directory: $MountPoint" >&2

Review Comment:
   The script creates the mount point directory (`mkdir -p`) before verifying that the path is actually mounted. If the shared filesystem is not mounted, subsequent heartbeat writes will succeed on the local filesystem, which can mask a storage outage and prevent HA fencing logic from triggering. Consider verifying the path is a real mount (e.g., `mountpoint -q` / `findmnt -M`) and returning a non-zero exit status in write mode when it is not mounted (and avoid creating the directory in that case).
   ```suggestion
   # Ensure mount point is valid; in write mode require it to be a real mount
   if [ ! -d "$MountPoint" ]; then
     echo "Mount point directory does not exist: $MountPoint" >&2
     exit 1
   fi
   
   # In write mode (default, rflag=0) verify that the path is an actual mount.
   # This avoids writing heartbeats to a local directory when shared storage is down.
   if [ "$rflag" -ne 1 ]; then
     mounted_ok=1
     if command -v mountpoint >/dev/null 2>&1; then
       mountpoint -q "$MountPoint" >/dev/null 2>&1
       mounted_ok=$?
     elif command -v findmnt >/dev/null 2>&1; then
       findmnt -M "$MountPoint" >/dev/null 2>&1
       mounted_ok=$?
     else
       # If neither mountpoint nor findmnt is available, fall back to directory check only.
       mounted_ok=0
     fi
   
     if [ "$mounted_ok" -ne 0 ]; then
       echo "Mount point is not a mounted filesystem: $MountPoint" >&2
   ```



##########
scripts/vm/hypervisor/kvm/kvmsmpheartbeat.sh:
##########
@@ -0,0 +1,205 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+help() {
+  printf "Usage: $0
+                -i identifier (required for CLI compatibility; value ignored by local-only heartbeat)
+                -p path (required for CLI compatibility; value ignored by local-only heartbeat)
+                -m mount point (local path where heartbeat will be written)
+                -h host (host IP/name to include in heartbeat filename)
+                -r write/read hb log (read-check mode)
+                -c cleanup (trigger emergency reboot)
+                -t interval between read hb log\n"
+  exit 1
+}
+
+#set -x
+NfsSvrIP=
+NfsSvrPath=
+MountPoint=
+HostIP=
+interval=
+rflag=0
+cflag=0
+
+while getopts 'i:p:m:h:t:rc' OPTION
+do
+  case $OPTION in
+  i)
+     NfsSvrIP="$OPTARG"
+     ;; # retained for CLI compatibility but unused for this script
+  p)
+     NfsSvrPath="$OPTARG"
+     ;; # retained for CLI compatibility but unused for this script
+  m)
+     MountPoint="$OPTARG"
+     ;;
+  h)
+     HostIP="$OPTARG"
+     ;;
+  r)
+     rflag=1
+     ;;
+  t)
+     interval="$OPTARG"
+     ;;
+  c)
+    cflag=1
+     ;;
+  *)
+     help
+     ;;
+  esac
+done
+
+# For heartbeat we require a mountpoint
+if [ -z "$MountPoint" ]
+then
+   echo "Mount point (-m) is required"
+   help
+fi
+
+# Ensure mount point exists and is writable
+if [ ! -d "$MountPoint" ]; then
+  mkdir -p "$MountPoint" 2>/dev/null
+  if [ $? -ne 0 ]; then
+    echo "Failed to create mount point directory: $MountPoint" >&2
+    exit 1
+  fi
+fi
+
+#delete VMs on this mountpoint (best-effort)
+deleteVMs() {
+  local mountPoint=$1
+  vmPids=$(ps aux | grep qemu | grep "$mountPoint" | awk '{print $2}' 2> /dev/null)
+
+  if [ -z "$vmPids" ]
+  then
+     return
+  fi
+
+  for pid in $vmPids
+  do
+     kill -9 $pid &> /dev/null
+  done
+}
+
+#checking is there the mount point present under $MountPoint?
+if grep -q "^[^ ]\+ $MountPoint " /proc/mounts
+then
+   # mount exists; if not in read-check mode, consider deleting VMs similar to original behavior
+   if [ "$rflag" == "0" ]
+   then
+     deleteVMs $MountPoint
+   fi
+else
+   # mount point not present — we don't remount in local-only script
+   # nothing to do here; keep for compatibility with original flow
+   :

Review Comment:
   When the mount is not present, this branch does nothing and the script continues, which means it can still write/read heartbeat files under an unmounted local directory. For HA safety, missing mount should be treated as a hard failure (exit non-zero) so the caller can retry/fence instead of operating on local disk.
   ```suggestion
      # mount point not present — treat as hard failure to avoid using local disk for HA heartbeat
      printf "Required mount point not present: %s\n" "$MountPoint" >&2
      exit 2
   ```



##########
scripts/vm/hypervisor/kvm/kvmsmpheartbeat.sh:
##########
@@ -0,0 +1,205 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+help() {
+  printf "Usage: $0
+                -i identifier (required for CLI compatibility; value ignored by local-only heartbeat)
+                -p path (required for CLI compatibility; value ignored by local-only heartbeat)
+                -m mount point (local path where heartbeat will be written)
+                -h host (host IP/name to include in heartbeat filename)
+                -r write/read hb log (read-check mode)
+                -c cleanup (trigger emergency reboot)
+                -t interval between read hb log\n"
+  exit 1
+}
+
+#set -x
+NfsSvrIP=
+NfsSvrPath=
+MountPoint=
+HostIP=
+interval=
+rflag=0
+cflag=0
+
+while getopts 'i:p:m:h:t:rc' OPTION
+do
+  case $OPTION in
+  i)
+     NfsSvrIP="$OPTARG"
+     ;; # retained for CLI compatibility but unused for this script
+  p)
+     NfsSvrPath="$OPTARG"
+     ;; # retained for CLI compatibility but unused for this script
+  m)
+     MountPoint="$OPTARG"
+     ;;
+  h)
+     HostIP="$OPTARG"
+     ;;
+  r)
+     rflag=1
+     ;;
+  t)
+     interval="$OPTARG"
+     ;;
+  c)
+    cflag=1
+     ;;
+  *)
+     help
+     ;;
+  esac
+done
+
+# For heartbeat we require a mountpoint
+if [ -z "$MountPoint" ]
+then
+   echo "Mount point (-m) is required"
+   help
+fi
+
+# Ensure mount point exists and is writable
+if [ ! -d "$MountPoint" ]; then
+  mkdir -p "$MountPoint" 2>/dev/null
+  if [ $? -ne 0 ]; then
+    echo "Failed to create mount point directory: $MountPoint" >&2
+    exit 1
+  fi
+fi
+
+#delete VMs on this mountpoint (best-effort)
+deleteVMs() {
+  local mountPoint=$1
+  vmPids=$(ps aux | grep qemu | grep "$mountPoint" | awk '{print $2}' 2> /dev/null)
+
+  if [ -z "$vmPids" ]
+  then
+     return
+  fi
+
+  for pid in $vmPids
+  do
+     kill -9 $pid &> /dev/null
+  done
+}
+
+#checking is there the mount point present under $MountPoint?
+if grep -q "^[^ ]\+ $MountPoint " /proc/mounts
+then
+   # mount exists; if not in read-check mode, consider deleting VMs similar to original behavior
+   if [ "$rflag" == "0" ]
+   then
+     deleteVMs $MountPoint
+   fi
+else
+   # mount point not present — we don't remount in local-only script
+   # nothing to do here; keep for compatibility with original flow
+   :
+fi
+
+hbFolder="$MountPoint/KVMHA/"

Review Comment:
   `hbFolder` already includes a trailing slash (`.../KVMHA/`), but `hbFile` adds another `/`, producing paths like `.../KVMHA//hb-...`. This works on Linux but is easy to overlook and can complicate log/debug output; consider constructing `hbFile` without the extra separator or removing the trailing slash from `hbFolder`.
   ```suggestion
   hbFolder="$MountPoint/KVMHA"
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KVM: Enable HA heartbeat on ShareMountPoint [cloudstack]

Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on PR #12773:
URL: https://github.com/apache/cloudstack/pull/12773#issuecomment-4136626509

   @blueorangutan package


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KVM: Enable HA heartbeat on ShareMountPoint [cloudstack]

Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on code in PR #12773:
URL: https://github.com/apache/cloudstack/pull/12773#discussion_r2923004353


##########
plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/KVMHAMonitor.java:
##########
@@ -34,6 +34,8 @@
 
 public class KVMHAMonitor extends KVMHABase implements Runnable {
 
+    public static final List<StoragePoolType> STORAGE_POOL_TYPES_WITH_HA_SUPPORT = List.of(StoragePoolType.NetworkFilesystem, StoragePoolType.SharedMountPoint);

Review Comment:
   the method `isPoolSupportHA` is already there.
   
   I will rename to LIBVIRT_STORAGE_POOL_TYPES_WITH_HA_SUPPORT



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KVM: Enable HA heartbeat on ShareMountPoint [cloudstack]

Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #12773:
URL: https://github.com/apache/cloudstack/pull/12773#issuecomment-4137264499

   @weizhouapache a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KVM: Enable HA heartbeat on ShareMountPoint [cloudstack]

Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on PR #12773:
URL: https://github.com/apache/cloudstack/pull/12773#issuecomment-4046356688

   thanks @DaanHoogland @sureshanaparti , and Copilot for review
   
   @NuxRo 
   I think this is ready for testing now


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KVM: Enable HA heartbeat on ShareMountPoint [cloudstack]

Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on PR #12773:
URL: https://github.com/apache/cloudstack/pull/12773#issuecomment-4024730438

   
   @blueoranguran package
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KVM: Enable HA heartbeat on ShareMountPoint [cloudstack]

Posted by "sureshanaparti (via GitHub)" <gi...@apache.org>.
sureshanaparti merged PR #12773:
URL: https://github.com/apache/cloudstack/pull/12773


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KVM: Enable HA heartbeat on ShareMountPoint [cloudstack]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #12773:
URL: https://github.com/apache/cloudstack/pull/12773#discussion_r2923702052


##########
plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/LibvirtStoragePool.java:
##########
@@ -320,13 +321,24 @@ public void setDetails(Map<String, String> details) {
 
     @Override
     public boolean isPoolSupportHA() {
-        return type == StoragePoolType.NetworkFilesystem;
+        return HighAvailabilityManager.LIBVIRT_STORAGE_POOL_TYPES_WITH_HA_SUPPORT.contains(type);
     }
 
     public String getHearthBeatPath() {
-        if (type == StoragePoolType.NetworkFilesystem) {
+        if (StoragePoolType.NetworkFilesystem.equals(type)) {
             String kvmScriptsDir = AgentPropertiesFileHandler.getPropertyValue(AgentProperties.KVM_SCRIPTS_DIR);
-            return Script.findScript(kvmScriptsDir, "kvmheartbeat.sh");
+            String scriptPath = Script.findScript(kvmScriptsDir, "kvmheartbeat.sh");
+            if (scriptPath == null) {
+                throw new CloudRuntimeException("Unable to find heartbeat script 'kvmheartbeat.sh' in directory: " + kvmScriptsDir);
+            }
+            return scriptPath;
+        } else if (StoragePoolType.SharedMountPoint.equals(type)) {
+            String kvmScriptsDir = AgentPropertiesFileHandler.getPropertyValue(AgentProperties.KVM_SCRIPTS_DIR);
+            String scriptPath = Script.findScript(kvmScriptsDir, "kvmsmpheartbeat.sh");
+            if (scriptPath == null) {
+                throw new CloudRuntimeException("Unable to find heartbeat script 'kvmsmpheartbeat.sh' in directory: " + kvmScriptsDir);
+            }
+            return scriptPath;

Review Comment:
   `getHearthBeatPath()` now throws if `kvmsmpheartbeat.sh` is missing, but agent startup validation in `LibvirtComputingResource` currently only checks for `kvmheartbeat.sh`. This means a packaging/config issue for the new script will surface later at runtime (during HA heartbeat) rather than failing fast during agent configuration. Consider adding a startup-time check for `kvmsmpheartbeat.sh` (or making `getHearthBeatPath()` return a failure message in `createHeartBeatCommand` instead of throwing).



##########
plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/KVMStoragePoolManager.java:
##########
@@ -390,6 +390,7 @@ public KVMStoragePool createStoragePool(String name, String host, int port, Stri
     private synchronized KVMStoragePool createStoragePool(String name, String host, int port, String path, String userInfo, StoragePoolType type, Map<String, String> details, boolean primaryStorage) {
         StorageAdaptor adaptor = getStorageAdaptor(type);
         KVMStoragePool pool = adaptor.createStoragePool(name, host, port, path, userInfo, type, details, primaryStorage);
+        pool.setType(type);
 

Review Comment:
   `pool.setType(type)` here looks like a workaround for libvirt pools (e.g., DIR pools) being mapped back to `StoragePoolType.Filesystem` in `LibvirtStorageAdaptor.getStoragePool(...)`. To avoid having to remember to do this in individual call sites (only some wrappers now do), consider setting the type in `KVMStoragePoolManager.getStoragePool(StoragePoolType type, ...)` before returning the pool (i.e., enforce the requested CloudStack pool type centrally).



##########
scripts/vm/hypervisor/kvm/kvmsmpheartbeat.sh:
##########
@@ -0,0 +1,205 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+help() {
+  printf "Usage: $0
+                -i identifier (required for CLI compatibility; value ignored by local-only heartbeat)
+                -p path (required for CLI compatibility; value ignored by local-only heartbeat)
+                -m mount point (local path where heartbeat will be written)
+                -h host (host IP/name to include in heartbeat filename)
+                -r write/read hb log (read-check mode)
+                -c cleanup (trigger emergency reboot)
+                -t interval between read hb log\n"
+  exit 1
+}
+
+#set -x
+NfsSvrIP=
+NfsSvrPath=
+MountPoint=
+HostIP=
+interval=
+rflag=0
+cflag=0
+
+while getopts 'i:p:m:h:t:rc' OPTION
+do
+  case $OPTION in
+  i)
+     NfsSvrIP="$OPTARG"
+     ;; # retained for CLI compatibility but unused for this script
+  p)
+     NfsSvrPath="$OPTARG"
+     ;; # retained for CLI compatibility but unused for this script
+  m)
+     MountPoint="$OPTARG"
+     ;;
+  h)
+     HostIP="$OPTARG"
+     ;;
+  r)
+     rflag=1
+     ;;
+  t)
+     interval="$OPTARG"
+     ;;
+  c)
+    cflag=1
+     ;;
+  *)
+     help
+     ;;
+  esac
+done
+
+# For heartbeat we require a mountpoint
+if [ -z "$MountPoint" ]
+then
+   echo "Mount point (-m) is required"
+   help
+fi
+
+# Ensure mount point exists and is writable
+if [ ! -d "$MountPoint" ]; then
+  mkdir -p "$MountPoint" 2>/dev/null
+  if [ $? -ne 0 ]; then
+    echo "Failed to create mount point directory: $MountPoint" >&2
+    exit 1
+  fi
+fi
+
+#delete VMs on this mountpoint (best-effort)
+deleteVMs() {
+  local mountPoint=$1
+  vmPids=$(ps aux | grep qemu | grep "$mountPoint" | awk '{print $2}' 2> /dev/null)
+
+  if [ -z "$vmPids" ]
+  then
+     return
+  fi
+
+  for pid in $vmPids
+  do
+     kill -9 $pid &> /dev/null
+  done
+}
+
+#checking is there the mount point present under $MountPoint?
+if grep -q "^[^ ]\+ $MountPoint " /proc/mounts

Review Comment:
   The mount check uses a regex with an unescaped `$MountPoint` (e.g., `.` or `+` in the path would change the match), and relies on GNU `\+` in basic regex. To avoid false positives/negatives across distros and unusual mount paths, prefer a field-equality check (e.g., `awk '$2==mp {found=1} END{exit !found}' mp="$MountPoint" /proc/mounts`) or `findmnt -T "$MountPoint"`.
   ```suggestion
   if awk -v mp="$MountPoint" '$2 == mp { found=1; exit } END { exit !found }' /proc/mounts
   ```



##########
scripts/vm/hypervisor/kvm/kvmsmpheartbeat.sh:
##########
@@ -0,0 +1,205 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+help() {
+  printf "Usage: $0
+                -i identifier (required for CLI compatibility; value ignored by local-only heartbeat)
+                -p path (required for CLI compatibility; value ignored by local-only heartbeat)
+                -m mount point (local path where heartbeat will be written)
+                -h host (host IP/name to include in heartbeat filename)
+                -r write/read hb log (read-check mode)
+                -c cleanup (trigger emergency reboot)
+                -t interval between read hb log\n"
+  exit 1
+}
+
+#set -x
+NfsSvrIP=
+NfsSvrPath=
+MountPoint=
+HostIP=
+interval=
+rflag=0
+cflag=0
+
+while getopts 'i:p:m:h:t:rc' OPTION
+do
+  case $OPTION in
+  i)
+     NfsSvrIP="$OPTARG"
+     ;; # retained for CLI compatibility but unused for this script
+  p)
+     NfsSvrPath="$OPTARG"
+     ;; # retained for CLI compatibility but unused for this script
+  m)
+     MountPoint="$OPTARG"
+     ;;
+  h)
+     HostIP="$OPTARG"
+     ;;
+  r)
+     rflag=1
+     ;;
+  t)
+     interval="$OPTARG"
+     ;;
+  c)
+    cflag=1
+     ;;
+  *)
+     help
+     ;;
+  esac
+done
+
+# For heartbeat we require a mountpoint
+if [ -z "$MountPoint" ]
+then
+   echo "Mount point (-m) is required"
+   help
+fi
+
+# Ensure mount point exists and is writable
+if [ ! -d "$MountPoint" ]; then
+  mkdir -p "$MountPoint" 2>/dev/null
+  if [ $? -ne 0 ]; then
+    echo "Failed to create mount point directory: $MountPoint" >&2
+    exit 1
+  fi

Review Comment:
   Creating the mount-point directory (`mkdir -p "$MountPoint"`) before verifying the shared filesystem is actually mounted can mask storage outages: the script can end up writing the heartbeat to the local root filesystem if the mount is missing. This can prevent the agent from detecting a heartbeat write failure (and therefore from fencing/rebooting when configured). Consider validating that `$MountPoint` is a mount target (e.g., via `findmnt -T` or exact field match in `/proc/mounts`) before creating/using the directory, and fail with non-zero exit when it isn’t mounted.
   ```suggestion
   # Ensure mount point exists and is a mounted filesystem
   if [ ! -d "$MountPoint" ]; then
     echo "Mount point directory does not exist: $MountPoint" >&2
     exit 1
   fi
   
   # Verify that $MountPoint is an active mount point
   if command -v findmnt >/dev/null 2>&1; then
     # Use findmnt to ensure that the mount target exactly matches $MountPoint
     if ! findmnt -n -o TARGET -T "$MountPoint" 2>/dev/null | grep -Fx -- "$MountPoint" >/dev/null 2>&1; then
       echo "Mount point is not a mounted filesystem: $MountPoint" >&2
       exit 1
     fi
   else
     # Fallback: check /proc/mounts for an exact mountpoint match
     if ! awk -v m="$MountPoint" '$2 == m {found=1} END {exit !found}' /proc/mounts; then
       echo "Mount point is not a mounted filesystem: $MountPoint" >&2
       exit 1
     fi
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KVM: Enable HA heartbeat on ShareMountPoint [cloudstack]

Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on code in PR #12773:
URL: https://github.com/apache/cloudstack/pull/12773#discussion_r2912817592


##########
plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/KVMStoragePoolManager.java:
##########
@@ -57,7 +57,7 @@ public class KVMStoragePoolManager {
     private final Map<String, StoragePoolInformation> _storagePools = new ConcurrentHashMap<String, StoragePoolInformation>();
     private final Map<String, StorageAdaptor> _storageMapper = new HashMap<String, StorageAdaptor>();
 
-    private StorageAdaptor getStorageAdaptor(StoragePoolType type) {
+    public StorageAdaptor getStorageAdaptor(StoragePoolType type) {

Review Comment:
   ```suggestion
       private StorageAdaptor getStorageAdaptor(StoragePoolType type) {
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KVM: Enable HA heartbeat on ShareMountPoint [cloudstack]

Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on code in PR #12773:
URL: https://github.com/apache/cloudstack/pull/12773#discussion_r2912833124


##########
plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/KVMStoragePoolManager.java:
##########
@@ -57,7 +57,7 @@ public class KVMStoragePoolManager {
     private final Map<String, StoragePoolInformation> _storagePools = new ConcurrentHashMap<String, StoragePoolInformation>();
     private final Map<String, StorageAdaptor> _storageMapper = new HashMap<String, StorageAdaptor>();
 
-    private StorageAdaptor getStorageAdaptor(StoragePoolType type) {
+    public StorageAdaptor getStorageAdaptor(StoragePoolType type) {

Review Comment:
   reverted @DaanHoogland 
   
   the change was part of a different solution which did not work well. I have reverted other changes but forgot to revert this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KVM: Enable HA heartbeat on ShareMountPoint [cloudstack]

Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on PR #12773:
URL: https://github.com/apache/cloudstack/pull/12773#issuecomment-4023462790

   @blueoranguran package


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KVM: Enable HA heartbeat on ShareMountPoint [cloudstack]

Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on code in PR #12773:
URL: https://github.com/apache/cloudstack/pull/12773#discussion_r2912855462


##########
scripts/vm/hypervisor/kvm/kvmsmpheartbeat.sh:
##########
@@ -0,0 +1,218 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+help() {
+  printf "Usage: $0
+                -i identifier (ignored for local-only heartbeat)
+                -p path (ignored for local-only heartbeat)
+                -m mount point (local path where heartbeat will be written)
+                -h host (host IP/name to include in heartbeat filename)
+                -r write/read hb log (read-check mode)
+                -c cleanup (trigger emergency reboot)
+                -t interval between read hb log\n"
+  exit 1
+}
+
+#set -x
+NfsSvrIP=
+NfsSvrPath=
+MountPoint=
+HostIP=
+interval=
+rflag=0
+cflag=0
+
+while getopts 'i:p:m:h:t:rc' OPTION
+do
+  case $OPTION in
+  i)
+     NfsSvrIP="$OPTARG"
+     ;; # retained for CLI compatibility but unused for local-only script
+  p)
+     NfsSvrPath="$OPTARG"
+     ;; # retained for CLI compatibility but unused for local-only script
+  m)
+     MountPoint="$OPTARG"
+     ;;
+  h)
+     HostIP="$OPTARG"
+     ;;
+  r)
+     rflag=1
+     ;;
+  t)
+     interval="$OPTARG"
+     ;;
+  c)
+    cflag=1
+     ;;
+  *)
+     help
+     ;;
+  esac
+done
+
+# Match original kvmheartbeat.sh: require NfsSvrIP parameter for CLI compatibility
+if [ -z "$NfsSvrIP" ]
+then
+   exit 1
+fi
+
+# For local-only heartbeat we require a mountpoint
+if [ -z "$MountPoint" ]
+then
+   echo "Mount point (-m) is required"
+   help
+fi
+
+# Ensure mount point exists and is writable
+if [ ! -d "$MountPoint" ]; then
+  mkdir -p "$MountPoint" 2>/dev/null
+  if [ $? -ne 0 ]; then
+    echo "Failed to create mount point directory: $MountPoint" >&2
+    exit 1
+  fi
+fi
+
+# Determine a sensible HostIP if not provided
+if [ -z "$HostIP" ]; then
+  # try to get a non-loopback IPv4 address, fallback to hostname
+  ipaddr=$(hostname -I 2>/dev/null | awk '{print $1}')
+  if [ -n "$ipaddr" ]; then
+    HostIP="$ipaddr"
+  else
+    HostIP=$(hostname)
+  fi
+fi
+
+#delete VMs on this mountpoint (best-effort)
+deleteVMs() {
+  local mountPoint=$1
+  vmPids=$(ps aux | grep qemu | grep "$mountPoint" | awk '{print $2}' 2> /dev/null)
+  if [ $? -gt 0 ]
+  then
+     return
+  fi
+
+  if [ -z "$vmPids" ]
+  then
+     return
+  fi
+
+  for pid in $vmPids
+  do
+     kill -9 $pid &> /dev/null
+  done
+}
+
+#checking is there the mount point present under $MountPoint?
+mounts=$(cat /proc/mounts | grep "$MountPoint")
+if [ $? -gt 0 ]
+then
+   # mount point not present — we don't remount in local-only script
+   # nothing to do here; keep for compatibility with original flow
+   :
+else

Review Comment:
   ```suggestion
   mounts=$(cat /proc/mounts | grep "$MountPoint ")
   if [ $? -gt 0 ]
   then
      # mount point not present — we don't remount in local-only script
      # nothing to do here; keep for compatibility with original flow
      :
   else
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KVM: Enable HA heartbeat on ShareMountPoint [cloudstack]

Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #12773:
URL: https://github.com/apache/cloudstack/pull/12773#issuecomment-4025314224

   Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✖️ debian ✔️ suse15. SL-JID 17048


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KVM: Enable HA heartbeat on ShareMountPoint [cloudstack]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #12773:
URL: https://github.com/apache/cloudstack/pull/12773#discussion_r2923909905


##########
scripts/vm/hypervisor/kvm/kvmsmpheartbeat.sh:
##########
@@ -0,0 +1,205 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+help() {
+  printf "Usage: $0
+                -i identifier (required for CLI compatibility; value ignored by local-only heartbeat)
+                -p path (required for CLI compatibility; value ignored by local-only heartbeat)
+                -m mount point (local path where heartbeat will be written)
+                -h host (host IP/name to include in heartbeat filename)
+                -r write/read hb log (read-check mode)
+                -c cleanup (trigger emergency reboot)
+                -t interval between read hb log\n"
+  exit 1
+}
+
+#set -x
+NfsSvrIP=
+NfsSvrPath=
+MountPoint=
+HostIP=
+interval=
+rflag=0
+cflag=0
+
+while getopts 'i:p:m:h:t:rc' OPTION
+do
+  case $OPTION in
+  i)
+     NfsSvrIP="$OPTARG"
+     ;; # retained for CLI compatibility but unused for this script
+  p)
+     NfsSvrPath="$OPTARG"
+     ;; # retained for CLI compatibility but unused for this script
+  m)
+     MountPoint="$OPTARG"
+     ;;
+  h)
+     HostIP="$OPTARG"
+     ;;
+  r)
+     rflag=1
+     ;;
+  t)
+     interval="$OPTARG"
+     ;;
+  c)
+    cflag=1
+     ;;
+  *)
+     help
+     ;;
+  esac
+done
+
+# For heartbeat we require a mountpoint
+if [ -z "$MountPoint" ]
+then
+   echo "Mount point (-m) is required"
+   help
+fi
+
+# Ensure mount point exists and is writable
+if [ ! -d "$MountPoint" ]; then
+  mkdir -p "$MountPoint" 2>/dev/null
+  if [ $? -ne 0 ]; then
+    echo "Failed to create mount point directory: $MountPoint" >&2
+    exit 1
+  fi
+fi
+

Review Comment:
   Creating the mount point directory here can mask a misconfiguration or missing mount: the script may end up writing heartbeat files on the local filesystem when the shared mount is not actually mounted. Consider requiring the mount point directory to pre-exist (or at least verifying it is a mounted filesystem) instead of `mkdir -p` in the heartbeat script.
   ```suggestion
   # Validate mount point exists, is (if possible) a mounted filesystem, and is writable
   if [ ! -d "$MountPoint" ]; then
     echo "Mount point directory does not exist: $MountPoint" >&2
     exit 1
   fi
   
   # If the 'mountpoint' utility is available, ensure this is an actual mount
   if command -v mountpoint >/dev/null 2>&1; then
     if ! mountpoint -q "$MountPoint"; then
       echo "Mount point is not a mounted filesystem: $MountPoint" >&2
       exit 1
     fi
   fi
   
   # Ensure the mount point is writable
   if [ ! -w "$MountPoint" ]; then
     echo "Mount point is not writable: $MountPoint" >&2
     exit 1
   fi
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KVM: Enable HA heartbeat on ShareMountPoint [cloudstack]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #12773:
URL: https://github.com/apache/cloudstack/pull/12773#discussion_r2924150004


##########
scripts/vm/hypervisor/kvm/kvmsmpheartbeat.sh:
##########
@@ -0,0 +1,218 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+help() {
+  printf "Usage: $0
+                -i identifier (required for CLI compatibility; value ignored by local-only heartbeat)
+                -p path (required for CLI compatibility; value ignored by local-only heartbeat)
+                -m mount point (local path where heartbeat will be written)
+                -h host (host IP/name to include in heartbeat filename)
+                -r write/read hb log (read-check mode)
+                -c cleanup (trigger emergency reboot)
+                -t interval between read hb log\n"
+  exit 1
+}
+
+#set -x
+NfsSvrIP=
+NfsSvrPath=
+MountPoint=
+HostIP=
+interval=
+rflag=0
+cflag=0
+
+while getopts 'i:p:m:h:t:rc' OPTION
+do
+  case $OPTION in
+  i)
+     NfsSvrIP="$OPTARG"
+     ;; # retained for CLI compatibility but unused for this script
+  p)
+     NfsSvrPath="$OPTARG"
+     ;; # retained for CLI compatibility but unused for this script
+  m)
+     MountPoint="$OPTARG"
+     ;;
+  h)
+     HostIP="$OPTARG"
+     ;;
+  r)
+     rflag=1
+     ;;
+  t)
+     interval="$OPTARG"
+     ;;
+  c)
+    cflag=1
+     ;;
+  *)
+     help
+     ;;
+  esac
+done
+
+# For heartbeat we require a mountpoint
+if [ -z "$MountPoint" ]
+then
+   echo "Mount point (-m) is required"
+   help
+fi
+
+# Validate mount point exists, is (if possible) a mounted filesystem, and is writable
+if [ ! -d "$MountPoint" ]; then
+  echo "Mount point directory does not exist: $MountPoint" >&2
+  exit 1
+fi
+
+# If the 'mountpoint' utility is available, ensure this is an actual mount
+if command -v mountpoint >/dev/null 2>&1; then
+  if ! mountpoint -q "$MountPoint"; then
+    echo "Mount point is not a mounted filesystem: $MountPoint" >&2
+    exit 1
+  fi
+fi
+
+# Ensure the mount point is writable
+if [ ! -w "$MountPoint" ]; then
+  echo "Mount point is not writable: $MountPoint" >&2
+  exit 1
+fi
+#delete VMs on this mountpoint (best-effort)
+deleteVMs() {
+  local mountPoint=$1
+  # ensure it ends with a single trailing slash
+  mountPoint="${mountPoint%/}/"
+
+  vmPids=$(ps aux | grep qemu | grep "$mountPoint" | awk '{print $2}' 2> /dev/null)
+
+  if [ -z "$vmPids" ]
+  then
+     return
+  fi
+
+  for pid in $vmPids
+  do
+     kill -9 $pid &> /dev/null
+  done
+}
+
+#checking is there the mount point present under $MountPoint?
+if grep -q "^[^ ]\+ $MountPoint " /proc/mounts
+then
+   # mount exists; nothing to do here; keep for compatibility with original flow
+   :
+else
+   # mount point not present
+   # if not in read-check mode, consider deleting VMs similar to original behavior
+   if [ "$rflag" == "0" ]
+   then
+     deleteVMs $MountPoint
+   fi
+fi
+
+hbFolder="$MountPoint/KVMHA"
+hbFile="$hbFolder/hb-$HostIP"
+
+write_hbLog() {
+#write the heart beat log
+  stat "$hbFile" &> /dev/null
+  if [ $? -gt 0 ]
+  then
+     # create a new one
+     mkdir -p "$hbFolder" &> /dev/null
+     # touch will be done by atomic write below; ensure folder is writable
+     if [ ! -w "$hbFolder" ]; then
+       printf "Folder not writable: $hbFolder" >&2
+       return 2
+     fi
+  fi
+
+  timestamp=$(date +%s)
+  # Write atomically to avoid partial writes (write to tmp then mv)
+  tmpfile="${hbFile}.$$"
+  printf "%s\n" "$timestamp" > "$tmpfile" 2>/dev/null
+  if [ $? -ne 0 ]; then
+    printf "Failed to write heartbeat to $tmpfile" >&2
+    return 2
+  fi
+  mv -f "$tmpfile" "$hbFile" 2>/dev/null
+  return $?
+}
+
+check_hbLog() {
+  hb_diff=0
+  if [ ! -f "$hbFile" ]; then
+    # signal large difference if file missing
+    hb_diff=999999
+    return 1
+  fi
+  now=$(date +%s)
+  hb=$(cat "$hbFile" 2>/dev/null)
+  if [ -z "$hb" ]; then
+    hb_diff=999998
+    return 1
+  fi
+  diff=`expr $now - $hb 2>/dev/null`
+  if [ $? -ne 0 ]
+  then
+    hb_diff=999997
+    return 1
+  fi
+  if [ -z "$interval" ]; then
+    # if no interval provided, consider 0 as success
+    if [ $diff -gt 0 ]; then
+      hb_diff=$diff
+      return 1
+    else
+      hb_diff=0
+      return 0
+    fi

Review Comment:
   The `-t` (interval) empty-path logic contradicts the comment and will almost always mark the host as DEAD because `diff` will normally be > 0. Either make `-t` mandatory for read-check mode (`-r`) and fail fast when missing, or treat a missing interval as a success case (e.g., only validate that the file contains a valid timestamp, optionally returning `hb_diff=$diff` without failing).
   ```suggestion
       # if no interval provided, treat any valid timestamp as success;
       # just report the time difference without failing based on staleness
       hb_diff=$diff
       return 0
   ```



##########
scripts/vm/hypervisor/kvm/kvmsmpheartbeat.sh:
##########
@@ -0,0 +1,218 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+help() {
+  printf "Usage: $0
+                -i identifier (required for CLI compatibility; value ignored by local-only heartbeat)
+                -p path (required for CLI compatibility; value ignored by local-only heartbeat)
+                -m mount point (local path where heartbeat will be written)
+                -h host (host IP/name to include in heartbeat filename)
+                -r write/read hb log (read-check mode)
+                -c cleanup (trigger emergency reboot)
+                -t interval between read hb log\n"
+  exit 1
+}
+
+#set -x
+NfsSvrIP=
+NfsSvrPath=
+MountPoint=
+HostIP=
+interval=
+rflag=0
+cflag=0
+
+while getopts 'i:p:m:h:t:rc' OPTION
+do
+  case $OPTION in
+  i)
+     NfsSvrIP="$OPTARG"
+     ;; # retained for CLI compatibility but unused for this script
+  p)
+     NfsSvrPath="$OPTARG"
+     ;; # retained for CLI compatibility but unused for this script
+  m)
+     MountPoint="$OPTARG"
+     ;;
+  h)
+     HostIP="$OPTARG"
+     ;;
+  r)
+     rflag=1
+     ;;
+  t)
+     interval="$OPTARG"
+     ;;
+  c)
+    cflag=1
+     ;;
+  *)
+     help
+     ;;
+  esac
+done
+
+# For heartbeat we require a mountpoint
+if [ -z "$MountPoint" ]
+then
+   echo "Mount point (-m) is required"
+   help
+fi
+
+# Validate mount point exists, is (if possible) a mounted filesystem, and is writable
+if [ ! -d "$MountPoint" ]; then
+  echo "Mount point directory does not exist: $MountPoint" >&2
+  exit 1
+fi
+
+# If the 'mountpoint' utility is available, ensure this is an actual mount
+if command -v mountpoint >/dev/null 2>&1; then
+  if ! mountpoint -q "$MountPoint"; then
+    echo "Mount point is not a mounted filesystem: $MountPoint" >&2
+    exit 1
+  fi
+fi
+
+# Ensure the mount point is writable
+if [ ! -w "$MountPoint" ]; then
+  echo "Mount point is not writable: $MountPoint" >&2
+  exit 1
+fi
+#delete VMs on this mountpoint (best-effort)
+deleteVMs() {
+  local mountPoint=$1
+  # ensure it ends with a single trailing slash
+  mountPoint="${mountPoint%/}/"
+
+  vmPids=$(ps aux | grep qemu | grep "$mountPoint" | awk '{print $2}' 2> /dev/null)
+
+  if [ -z "$vmPids" ]
+  then
+     return
+  fi
+
+  for pid in $vmPids
+  do
+     kill -9 $pid &> /dev/null
+  done

Review Comment:
   This process selection is fragile (can match the grep pipeline itself and may miss qemu processes depending on cmdline format), and always uses `kill -9` (no grace period), which can increase corruption risk. Prefer a more reliable match (e.g., a single pattern match that avoids self-matching) and attempt a graceful shutdown first (SIGTERM with a short wait) before SIGKILL as a fallback.



##########
scripts/vm/hypervisor/kvm/kvmsmpheartbeat.sh:
##########
@@ -0,0 +1,218 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+help() {
+  printf "Usage: $0
+                -i identifier (required for CLI compatibility; value ignored by local-only heartbeat)
+                -p path (required for CLI compatibility; value ignored by local-only heartbeat)
+                -m mount point (local path where heartbeat will be written)
+                -h host (host IP/name to include in heartbeat filename)
+                -r write/read hb log (read-check mode)
+                -c cleanup (trigger emergency reboot)
+                -t interval between read hb log\n"
+  exit 1
+}
+
+#set -x
+NfsSvrIP=
+NfsSvrPath=
+MountPoint=
+HostIP=
+interval=
+rflag=0
+cflag=0
+
+while getopts 'i:p:m:h:t:rc' OPTION
+do
+  case $OPTION in
+  i)
+     NfsSvrIP="$OPTARG"
+     ;; # retained for CLI compatibility but unused for this script
+  p)
+     NfsSvrPath="$OPTARG"
+     ;; # retained for CLI compatibility but unused for this script
+  m)
+     MountPoint="$OPTARG"
+     ;;
+  h)
+     HostIP="$OPTARG"
+     ;;
+  r)
+     rflag=1
+     ;;
+  t)
+     interval="$OPTARG"
+     ;;
+  c)
+    cflag=1
+     ;;
+  *)
+     help
+     ;;
+  esac
+done
+
+# For heartbeat we require a mountpoint
+if [ -z "$MountPoint" ]
+then
+   echo "Mount point (-m) is required"
+   help
+fi
+
+# Validate mount point exists, is (if possible) a mounted filesystem, and is writable
+if [ ! -d "$MountPoint" ]; then
+  echo "Mount point directory does not exist: $MountPoint" >&2
+  exit 1
+fi
+
+# If the 'mountpoint' utility is available, ensure this is an actual mount
+if command -v mountpoint >/dev/null 2>&1; then
+  if ! mountpoint -q "$MountPoint"; then
+    echo "Mount point is not a mounted filesystem: $MountPoint" >&2
+    exit 1
+  fi
+fi
+
+# Ensure the mount point is writable
+if [ ! -w "$MountPoint" ]; then
+  echo "Mount point is not writable: $MountPoint" >&2
+  exit 1
+fi
+#delete VMs on this mountpoint (best-effort)
+deleteVMs() {
+  local mountPoint=$1
+  # ensure it ends with a single trailing slash
+  mountPoint="${mountPoint%/}/"
+
+  vmPids=$(ps aux | grep qemu | grep "$mountPoint" | awk '{print $2}' 2> /dev/null)
+
+  if [ -z "$vmPids" ]
+  then
+     return
+  fi
+
+  for pid in $vmPids
+  do
+     kill -9 $pid &> /dev/null
+  done
+}
+
+#checking is there the mount point present under $MountPoint?
+if grep -q "^[^ ]\+ $MountPoint " /proc/mounts
+then
+   # mount exists; nothing to do here; keep for compatibility with original flow
+   :
+else
+   # mount point not present
+   # if not in read-check mode, consider deleting VMs similar to original behavior
+   if [ "$rflag" == "0" ]
+   then
+     deleteVMs $MountPoint
+   fi
+fi
+
+hbFolder="$MountPoint/KVMHA"
+hbFile="$hbFolder/hb-$HostIP"
+
+write_hbLog() {
+#write the heart beat log
+  stat "$hbFile" &> /dev/null
+  if [ $? -gt 0 ]
+  then
+     # create a new one
+     mkdir -p "$hbFolder" &> /dev/null
+     # touch will be done by atomic write below; ensure folder is writable
+     if [ ! -w "$hbFolder" ]; then
+       printf "Folder not writable: $hbFolder" >&2
+       return 2
+     fi
+  fi
+
+  timestamp=$(date +%s)
+  # Write atomically to avoid partial writes (write to tmp then mv)
+  tmpfile="${hbFile}.$$"
+  printf "%s\n" "$timestamp" > "$tmpfile" 2>/dev/null
+  if [ $? -ne 0 ]; then
+    printf "Failed to write heartbeat to $tmpfile" >&2
+    return 2
+  fi

Review Comment:
   These `printf` error messages do not include a trailing newline, which can make logs harder to read/parse. Add `\\n` to both messages (and consider including context like mount point or hbFile) to improve operational readability.



##########
plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/LibvirtStoragePool.java:
##########
@@ -410,4 +422,8 @@ public Boolean vmActivityCheck(HAStoragePool pool, HostTO host, Duration activit
             return true;
         }
     }
+
+    public void setType(StoragePoolType type) {

Review Comment:
   Introducing a public setter for `type` makes `LibvirtStoragePool` mutable after construction. Since pools are commonly cached and accessed across threads, mutating `type` can lead to races/inconsistent HA behavior (e.g., `isPoolSupportHA()` or `getHearthBeatPath()` observing an unexpected or transient value). Prefer initializing `type` during pool construction (or via a single-threaded initialization path) and make it immutable; if mutation is unavoidable, enforce thread-safety (e.g., synchronization or `volatile`) and constrain visibility of the setter.
   ```suggestion
       void setType(StoragePoolType type) {
   ```



##########
scripts/vm/hypervisor/kvm/kvmsmpheartbeat.sh:
##########
@@ -0,0 +1,218 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+help() {
+  printf "Usage: $0
+                -i identifier (required for CLI compatibility; value ignored by local-only heartbeat)
+                -p path (required for CLI compatibility; value ignored by local-only heartbeat)
+                -m mount point (local path where heartbeat will be written)
+                -h host (host IP/name to include in heartbeat filename)
+                -r write/read hb log (read-check mode)
+                -c cleanup (trigger emergency reboot)
+                -t interval between read hb log\n"
+  exit 1
+}
+
+#set -x
+NfsSvrIP=
+NfsSvrPath=
+MountPoint=
+HostIP=
+interval=
+rflag=0
+cflag=0
+
+while getopts 'i:p:m:h:t:rc' OPTION
+do
+  case $OPTION in
+  i)
+     NfsSvrIP="$OPTARG"
+     ;; # retained for CLI compatibility but unused for this script
+  p)
+     NfsSvrPath="$OPTARG"
+     ;; # retained for CLI compatibility but unused for this script
+  m)
+     MountPoint="$OPTARG"
+     ;;
+  h)
+     HostIP="$OPTARG"
+     ;;
+  r)
+     rflag=1
+     ;;
+  t)
+     interval="$OPTARG"
+     ;;
+  c)
+    cflag=1
+     ;;
+  *)
+     help
+     ;;
+  esac
+done
+
+# For heartbeat we require a mountpoint
+if [ -z "$MountPoint" ]
+then
+   echo "Mount point (-m) is required"
+   help
+fi
+
+# Validate mount point exists, is (if possible) a mounted filesystem, and is writable
+if [ ! -d "$MountPoint" ]; then
+  echo "Mount point directory does not exist: $MountPoint" >&2
+  exit 1
+fi
+
+# If the 'mountpoint' utility is available, ensure this is an actual mount
+if command -v mountpoint >/dev/null 2>&1; then
+  if ! mountpoint -q "$MountPoint"; then
+    echo "Mount point is not a mounted filesystem: $MountPoint" >&2
+    exit 1
+  fi
+fi
+
+# Ensure the mount point is writable
+if [ ! -w "$MountPoint" ]; then
+  echo "Mount point is not writable: $MountPoint" >&2
+  exit 1
+fi
+#delete VMs on this mountpoint (best-effort)
+deleteVMs() {
+  local mountPoint=$1
+  # ensure it ends with a single trailing slash
+  mountPoint="${mountPoint%/}/"
+
+  vmPids=$(ps aux | grep qemu | grep "$mountPoint" | awk '{print $2}' 2> /dev/null)
+
+  if [ -z "$vmPids" ]
+  then
+     return
+  fi
+
+  for pid in $vmPids
+  do
+     kill -9 $pid &> /dev/null
+  done
+}
+
+#checking is there the mount point present under $MountPoint?
+if grep -q "^[^ ]\+ $MountPoint " /proc/mounts
+then
+   # mount exists; nothing to do here; keep for compatibility with original flow
+   :
+else
+   # mount point not present
+   # if not in read-check mode, consider deleting VMs similar to original behavior
+   if [ "$rflag" == "0" ]
+   then
+     deleteVMs $MountPoint
+   fi
+fi
+
+hbFolder="$MountPoint/KVMHA"
+hbFile="$hbFolder/hb-$HostIP"
+
+write_hbLog() {
+#write the heart beat log
+  stat "$hbFile" &> /dev/null
+  if [ $? -gt 0 ]
+  then
+     # create a new one
+     mkdir -p "$hbFolder" &> /dev/null
+     # touch will be done by atomic write below; ensure folder is writable
+     if [ ! -w "$hbFolder" ]; then
+       printf "Folder not writable: $hbFolder" >&2
+       return 2
+     fi

Review Comment:
   These `printf` error messages do not include a trailing newline, which can make logs harder to read/parse. Add `\\n` to both messages (and consider including context like mount point or hbFile) to improve operational readability.



##########
engine/components-api/src/main/java/com/cloud/ha/HighAvailabilityManager.java:
##########
@@ -32,6 +33,8 @@
  */
 public interface HighAvailabilityManager extends Manager {
 
+    List<StoragePoolType> LIBVIRT_STORAGE_POOL_TYPES_WITH_HA_SUPPORT = List.of(StoragePoolType.NetworkFilesystem, StoragePoolType.SharedMountPoint);

Review Comment:
   This constant is used as a membership test (`contains`) across the codebase. Representing it as a `Set<StoragePoolType>` (ideally `EnumSet`) makes intent clearer and avoids linear scans as the supported types list grows. Consider changing this to a `Set` and updating call sites accordingly.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KVM: Enable HA heartbeat on ShareMountPoint [cloudstack]

Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #12773:
URL: https://github.com/apache/cloudstack/pull/12773#issuecomment-4023577168

   ## [Codecov](https://app.codecov.io/gh/apache/cloudstack/pull/12773?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :white_check_mark: All modified and coverable lines are covered by tests.
   :white_check_mark: Project coverage is 3.70%. Comparing base ([`c3d6a8c`](https://app.codecov.io/gh/apache/cloudstack/commit/c3d6a8cff72c47412670a914e3f4d30ad06bc46f?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`93b95aa`](https://app.codecov.io/gh/apache/cloudstack/commit/93b95aa80ebac0cb5b850da44321a197176ce313?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   > :exclamation:  There is a different number of reports uploaded between BASE (c3d6a8c) and HEAD (93b95aa). Click for more details.
   > 
   > <details><summary>HEAD has 1 upload less than BASE</summary>
   >
   >| Flag | BASE (c3d6a8c) | HEAD (93b95aa) |
   >|------|------|------|
   >|unittests|1|0|
   ></details>
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@              Coverage Diff              @@
   ##               4.22   #12773       +/-   ##
   =============================================
   - Coverage     17.61%    3.70%   -13.91%     
   =============================================
     Files          5917      448     -5469     
     Lines        531402    38019   -493383     
     Branches      64971     7035    -57936     
   =============================================
   - Hits          93586     1409    -92177     
   + Misses       427262    36423   -390839     
   + Partials      10554      187    -10367     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/cloudstack/pull/12773/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [uitests](https://app.codecov.io/gh/apache/cloudstack/pull/12773/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `3.70% <ø> (ø)` | |
   | [unittests](https://app.codecov.io/gh/apache/cloudstack/pull/12773/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/cloudstack/pull/12773?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KVM: Enable HA heartbeat on ShareMountPoint [cloudstack]

Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #12773:
URL: https://github.com/apache/cloudstack/pull/12773#issuecomment-4046328768

   @weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KVM: Enable HA heartbeat on ShareMountPoint [cloudstack]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #12773:
URL: https://github.com/apache/cloudstack/pull/12773#discussion_r2923100207


##########
scripts/vm/hypervisor/kvm/kvmsmpheartbeat.sh:
##########
@@ -0,0 +1,212 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+help() {
+  printf "Usage: $0
+                -i identifier (required for CLI compatibility; value ignored by local-only heartbeat)
+                -p path (required for CLI compatibility; value ignored by local-only heartbeat)
+                -m mount point (local path where heartbeat will be written)
+                -h host (host IP/name to include in heartbeat filename)
+                -r write/read hb log (read-check mode)
+                -c cleanup (trigger emergency reboot)
+                -t interval between read hb log\n"
+  exit 1
+}
+
+#set -x
+NfsSvrIP=
+NfsSvrPath=
+MountPoint=
+HostIP=
+interval=
+rflag=0
+cflag=0
+
+while getopts 'i:p:m:h:t:rc' OPTION
+do
+  case $OPTION in
+  i)
+     NfsSvrIP="$OPTARG"
+     ;; # retained for CLI compatibility but unused for local-only script
+  p)
+     NfsSvrPath="$OPTARG"
+     ;; # retained for CLI compatibility but unused for local-only script
+  m)
+     MountPoint="$OPTARG"
+     ;;
+  h)
+     HostIP="$OPTARG"
+     ;;
+  r)
+     rflag=1
+     ;;
+  t)
+     interval="$OPTARG"
+     ;;
+  c)
+    cflag=1
+     ;;
+  *)
+     help
+     ;;
+  esac
+done
+
+# For local-only heartbeat we require a mountpoint
+if [ -z "$MountPoint" ]
+then
+   echo "Mount point (-m) is required"
+   help
+fi
+
+# Ensure mount point exists and is writable
+if [ ! -d "$MountPoint" ]; then
+  mkdir -p "$MountPoint" 2>/dev/null
+  if [ $? -ne 0 ]; then
+    echo "Failed to create mount point directory: $MountPoint" >&2
+    exit 1
+  fi
+fi
+
+# Determine a sensible HostIP if not provided
+if [ -z "$HostIP" ]; then
+  # try to get a non-loopback IPv4 address, fallback to hostname
+  ipaddr=$(hostname -I 2>/dev/null | awk '{print $1}')
+  if [ -n "$ipaddr" ]; then
+    HostIP="$ipaddr"
+  else
+    HostIP=$(hostname)
+  fi
+fi
+
+#delete VMs on this mountpoint (best-effort)
+deleteVMs() {
+  local mountPoint=$1
+  vmPids=$(ps aux | grep qemu | grep "$mountPoint" | awk '{print $2}' 2> /dev/null)
+  if [ $? -gt 0 ]
+  then
+     return
+  fi
+
+  if [ -z "$vmPids" ]
+  then
+     return
+  fi
+
+  for pid in $vmPids
+  do
+     kill -9 $pid &> /dev/null
+  done
+}
+
+#checking is there the mount point present under $MountPoint?
+mounts=$(cat /proc/mounts | grep "$MountPoint")
+if [ $? -gt 0 ]
+then
+   # mount point not present — we don't remount in local-only script
+   # nothing to do here; keep for compatibility with original flow
+   :
+else
+   # mount exists; if not in read-check mode, consider deleting VMs similar to original behavior
+   if [ "$rflag" == "0" ]
+   then
+     deleteVMs $MountPoint
+   fi
+fi
+

Review Comment:
   `deleteVMs` is executed when the mount *exists* (the `grep` succeeds) and the script is in write mode. In the original `kvmheartbeat.sh`, VMs are only killed after a remount attempt when the expected mount is missing. As written, this can kill running VMs on every successful heartbeat run whenever the storage is mounted. The condition should be inverted/adjusted so VM killing is only done for the intended failure/remount scenario (or removed entirely for the local-only heartbeat flow).
   ```suggestion
      # mount exists; in local-only heartbeat we do not kill VMs based on mount state
      # original kvmheartbeat.sh only deleted VMs after a failed remount; there is
      # no such remount logic here, so VM deletion is intentionally disabled
      :
   fi
   ```



##########
scripts/vm/hypervisor/kvm/kvmsmpheartbeat.sh:
##########
@@ -0,0 +1,212 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+help() {
+  printf "Usage: $0
+                -i identifier (required for CLI compatibility; value ignored by local-only heartbeat)
+                -p path (required for CLI compatibility; value ignored by local-only heartbeat)
+                -m mount point (local path where heartbeat will be written)
+                -h host (host IP/name to include in heartbeat filename)
+                -r write/read hb log (read-check mode)
+                -c cleanup (trigger emergency reboot)
+                -t interval between read hb log\n"
+  exit 1
+}
+
+#set -x
+NfsSvrIP=
+NfsSvrPath=
+MountPoint=
+HostIP=
+interval=
+rflag=0
+cflag=0
+
+while getopts 'i:p:m:h:t:rc' OPTION
+do
+  case $OPTION in
+  i)
+     NfsSvrIP="$OPTARG"
+     ;; # retained for CLI compatibility but unused for local-only script
+  p)
+     NfsSvrPath="$OPTARG"
+     ;; # retained for CLI compatibility but unused for local-only script
+  m)
+     MountPoint="$OPTARG"
+     ;;
+  h)
+     HostIP="$OPTARG"
+     ;;
+  r)
+     rflag=1
+     ;;
+  t)
+     interval="$OPTARG"
+     ;;
+  c)
+    cflag=1
+     ;;
+  *)
+     help
+     ;;
+  esac
+done
+
+# For local-only heartbeat we require a mountpoint
+if [ -z "$MountPoint" ]
+then
+   echo "Mount point (-m) is required"
+   help
+fi
+
+# Ensure mount point exists and is writable
+if [ ! -d "$MountPoint" ]; then
+  mkdir -p "$MountPoint" 2>/dev/null
+  if [ $? -ne 0 ]; then
+    echo "Failed to create mount point directory: $MountPoint" >&2
+    exit 1
+  fi
+fi
+
+# Determine a sensible HostIP if not provided
+if [ -z "$HostIP" ]; then
+  # try to get a non-loopback IPv4 address, fallback to hostname
+  ipaddr=$(hostname -I 2>/dev/null | awk '{print $1}')
+  if [ -n "$ipaddr" ]; then
+    HostIP="$ipaddr"
+  else
+    HostIP=$(hostname)
+  fi
+fi
+
+#delete VMs on this mountpoint (best-effort)
+deleteVMs() {
+  local mountPoint=$1
+  vmPids=$(ps aux | grep qemu | grep "$mountPoint" | awk '{print $2}' 2> /dev/null)
+  if [ $? -gt 0 ]
+  then
+     return
+  fi
+
+  if [ -z "$vmPids" ]
+  then
+     return
+  fi
+
+  for pid in $vmPids
+  do
+     kill -9 $pid &> /dev/null
+  done
+}
+
+#checking is there the mount point present under $MountPoint?
+mounts=$(cat /proc/mounts | grep "$MountPoint")
+if [ $? -gt 0 ]
+then
+   # mount point not present — we don't remount in local-only script
+   # nothing to do here; keep for compatibility with original flow
+   :
+else
+   # mount exists; if not in read-check mode, consider deleting VMs similar to original behavior
+   if [ "$rflag" == "0" ]
+   then
+     deleteVMs $MountPoint
+   fi

Review Comment:
   The mount-point detection uses `grep "$MountPoint"` on `/proc/mounts`, which can yield false positives when the mount point is a prefix of another path (e.g. `/mnt/cloudstack1` matching `/mnt/cloudstack10`). This can cause the script to treat the mount as present/absent incorrectly and trigger follow-on actions. Use an exact match on the mount target field (e.g. `findmnt -n -o TARGET --target "$MountPoint"` or matching the second column in `/proc/mounts`).
   ```suggestion
   if grep -q "^[^ ]\+ $MountPoint " /proc/mounts
   then
      # mount exists; if not in read-check mode, consider deleting VMs similar to original behavior
      if [ "$rflag" == "0" ]
      then
        deleteVMs $MountPoint
      fi
   else
      # mount point not present — we don't remount in local-only script
      # nothing to do here; keep for compatibility with original flow
      :
   ```



##########
scripts/vm/hypervisor/kvm/kvmsmpheartbeat.sh:
##########
@@ -0,0 +1,212 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+help() {
+  printf "Usage: $0
+                -i identifier (required for CLI compatibility; value ignored by local-only heartbeat)
+                -p path (required for CLI compatibility; value ignored by local-only heartbeat)
+                -m mount point (local path where heartbeat will be written)
+                -h host (host IP/name to include in heartbeat filename)
+                -r write/read hb log (read-check mode)
+                -c cleanup (trigger emergency reboot)
+                -t interval between read hb log\n"
+  exit 1
+}
+
+#set -x
+NfsSvrIP=
+NfsSvrPath=
+MountPoint=
+HostIP=
+interval=
+rflag=0
+cflag=0
+
+while getopts 'i:p:m:h:t:rc' OPTION
+do
+  case $OPTION in
+  i)
+     NfsSvrIP="$OPTARG"
+     ;; # retained for CLI compatibility but unused for local-only script
+  p)
+     NfsSvrPath="$OPTARG"
+     ;; # retained for CLI compatibility but unused for local-only script
+  m)
+     MountPoint="$OPTARG"
+     ;;
+  h)
+     HostIP="$OPTARG"
+     ;;
+  r)
+     rflag=1
+     ;;
+  t)
+     interval="$OPTARG"
+     ;;
+  c)
+    cflag=1
+     ;;
+  *)
+     help
+     ;;
+  esac
+done
+
+# For local-only heartbeat we require a mountpoint
+if [ -z "$MountPoint" ]
+then
+   echo "Mount point (-m) is required"
+   help
+fi
+
+# Ensure mount point exists and is writable
+if [ ! -d "$MountPoint" ]; then
+  mkdir -p "$MountPoint" 2>/dev/null
+  if [ $? -ne 0 ]; then
+    echo "Failed to create mount point directory: $MountPoint" >&2
+    exit 1
+  fi
+fi
+
+# Determine a sensible HostIP if not provided
+if [ -z "$HostIP" ]; then
+  # try to get a non-loopback IPv4 address, fallback to hostname
+  ipaddr=$(hostname -I 2>/dev/null | awk '{print $1}')
+  if [ -n "$ipaddr" ]; then
+    HostIP="$ipaddr"
+  else
+    HostIP=$(hostname)
+  fi
+fi
+
+#delete VMs on this mountpoint (best-effort)
+deleteVMs() {
+  local mountPoint=$1
+  vmPids=$(ps aux | grep qemu | grep "$mountPoint" | awk '{print $2}' 2> /dev/null)
+  if [ $? -gt 0 ]
+  then
+     return
+  fi
+
+  if [ -z "$vmPids" ]
+  then
+     return
+  fi
+
+  for pid in $vmPids
+  do
+     kill -9 $pid &> /dev/null
+  done
+}
+
+#checking is there the mount point present under $MountPoint?
+mounts=$(cat /proc/mounts | grep "$MountPoint")
+if [ $? -gt 0 ]
+then
+   # mount point not present — we don't remount in local-only script
+   # nothing to do here; keep for compatibility with original flow
+   :
+else
+   # mount exists; if not in read-check mode, consider deleting VMs similar to original behavior
+   if [ "$rflag" == "0" ]
+   then
+     deleteVMs $MountPoint
+   fi
+fi
+
+hbFolder="$MountPoint/KVMHA/"
+hbFile="$hbFolder/hb-$HostIP"
+
+write_hbLog() {
+#write the heart beat log
+  stat "$hbFile" &> /dev/null
+  if [ $? -gt 0 ]
+  then
+     # create a new one
+     mkdir -p "$hbFolder" &> /dev/null
+     # touch will be done by atomic write below; ensure folder is writable
+     if [ ! -w "$hbFolder" ]; then
+       printf "Folder not writable: $hbFolder" >&2
+       return 2
+     fi
+  fi
+
+  timestamp=$(date +%s)
+  # Write atomically to avoid partial writes (write to tmp then mv)
+  tmpfile="${hbFile}.$$"
+  printf "%s\n" "$timestamp" > "$tmpfile" 2>/dev/null
+  if [ $? -ne 0 ]; then
+    printf "Failed to write heartbeat to $tmpfile" >&2
+    return 2
+  fi
+  mv -f "$tmpfile" "$hbFile" 2>/dev/null
+  return $?
+}
+
+check_hbLog() {
+  if [ ! -f "$hbFile" ]; then
+    # signal large difference if file missing
+    return 999999
+  fi
+  now=$(date +%s)
+  hb=$(cat "$hbFile" 2>/dev/null)
+  if [ -z "$hb" ]; then
+    return 999998
+  fi
+  diff=`expr $now - $hb 2>/dev/null`
+  if [ $? -ne 0 ]
+  then
+    return 999997
+  fi
+  if [ -z "$interval" ]; then
+    # if no interval provided, consider 0 as success
+    if [ $diff -gt 0 ]; then
+      return $diff
+    else
+      return 0
+    fi
+  fi
+  if [ $diff -gt $interval ]
+  then
+    return $diff
+  fi
+  return 0
+}
+
+if [ "$rflag" == "1" ]
+then
+  check_hbLog
+  diff=$?
+  if [ $diff == 0 ]

Review Comment:
   `check_hbLog` returns large numeric values (e.g. `return 999999` / `return $diff`) and the caller reads them via `$?`. In bash, function return/exit codes are limited to 0-255, so values wrap modulo 256 and can incorrectly report ALIVE (e.g. a diff of 512 seconds becomes exit code 0). Use stdout to pass the diff (or store it in a variable) and use the exit status purely as a boolean (alive/dead).
   ```suggestion
       echo 999999
       return 1
     fi
     now=$(date +%s)
     hb=$(cat "$hbFile" 2>/dev/null)
     if [ -z "$hb" ]; then
       echo 999998
       return 1
     fi
     diff=`expr $now - $hb 2>/dev/null`
     if [ $? -ne 0 ]
     then
       echo 999997
       return 1
     fi
     if [ -z "$interval" ]; then
       # if no interval provided, consider 0 as success
       if [ $diff -gt 0 ]; then
         echo "$diff"
         return 1
       else
         echo 0
         return 0
       fi
     fi
     if [ $diff -gt $interval ]
     then
       echo "$diff"
       return 1
     fi
     echo 0
     return 0
   }
   
   if [ "$rflag" == "1" ]
   then
     diff=$(check_hbLog)
     status=$?
     if [ "$status" -eq 0 ]
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KVM: Enable HA heartbeat on ShareMountPoint [cloudstack]

Posted by "sureshanaparti (via GitHub)" <gi...@apache.org>.
sureshanaparti commented on PR #12773:
URL: https://github.com/apache/cloudstack/pull/12773#issuecomment-4045202925

   @weizhouapache any doc update required for this?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KVM: Enable HA heartbeat on ShareMountPoint [cloudstack]

Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on PR #12773:
URL: https://github.com/apache/cloudstack/pull/12773#issuecomment-4137260276

   @blueorangutan test


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KVM: Enable HA heartbeat on ShareMountPoint [cloudstack]

Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #12773:
URL: https://github.com/apache/cloudstack/pull/12773#issuecomment-4141908162

   <b>[SF] Trillian test result (tid-15749)</b>
   Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
   Total time taken: 57842 seconds
   Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr12773-t15749-kvm-ol8.zip
   Smoke tests completed. 148 look OK, 1 have errors, 0 did not run
   Only failed and skipped tests results shown below:
   
   
   Test | Result | Time (s) | Test File
   --- | --- | --- | ---
   test_01_deployVMInSharedNetwork | `Failure` | 331.05 | test_network.py
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KVM: Enable HA heartbeat on ShareMountPoint [cloudstack]

Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #12773:
URL: https://github.com/apache/cloudstack/pull/12773#issuecomment-4025831060

   @weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KVM: Enable HA heartbeat on ShareMountPoint [cloudstack]

Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on PR #12773:
URL: https://github.com/apache/cloudstack/pull/12773#issuecomment-4029357494

   @sureshanaparti @rajujith 
   added this to 4.22.1 milestone, a small improvement but may benefit several users
   
   cc @DaanHoogland @NuxRo 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KVM: Enable HA heartbeat on ShareMountPoint [cloudstack]

Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #12773:
URL: https://github.com/apache/cloudstack/pull/12773#issuecomment-4046886375

   Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 17117


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KVM: Enable HA heartbeat on ShareMountPoint [cloudstack]

Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #12773:
URL: https://github.com/apache/cloudstack/pull/12773#issuecomment-4053180883

   <b>[SF] Trillian test result (tid-15631)</b>
   Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
   Total time taken: 58208 seconds
   Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr12773-t15631-kvm-ol8.zip
   Smoke tests completed. 146 look OK, 3 have errors, 0 did not run
   Only failed and skipped tests results shown below:
   
   
   Test | Result | Time (s) | Test File
   --- | --- | --- | ---
   ContextSuite context=TestListIdsParams>:teardown | `Error` | 1.14 | test_list_ids_parameter.py
   test_01_snapshot_root_disk | `Error` | 4.78 | test_snapshots.py
   test_02_list_snapshots_with_removed_data_store | `Error` | 47.49 | test_snapshots.py
   test_02_list_snapshots_with_removed_data_store | `Error` | 47.50 | test_snapshots.py
   ContextSuite context=TestSnapshotStandaloneBackup>:teardown | `Error` | 30.40 | test_snapshots.py
   test_01_snapshot_usage | `Error` | 34.85 | test_usage.py
   test_01_vpn_usage | `Error` | 1.08 | test_usage.py
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KVM: Enable HA heartbeat on ShareMountPoint [cloudstack]

Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on PR #12773:
URL: https://github.com/apache/cloudstack/pull/12773#issuecomment-4026855652

   @blueorangutan test


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KVM: Enable HA heartbeat on ShareMountPoint [cloudstack]

Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #12773:
URL: https://github.com/apache/cloudstack/pull/12773#issuecomment-4031302230

   <b>[SF] Trillian test result (tid-15586)</b>
   Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
   Total time taken: 55643 seconds
   Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr12773-t15586-kvm-ol8.zip
   Smoke tests completed. 146 look OK, 3 have errors, 0 did not run
   Only failed and skipped tests results shown below:
   
   
   Test | Result | Time (s) | Test File
   --- | --- | --- | ---
   ContextSuite context=TestListIdsParams>:teardown | `Error` | 1.12 | test_list_ids_parameter.py
   test_01_snapshot_root_disk | `Error` | 7.74 | test_snapshots.py
   test_02_list_snapshots_with_removed_data_store | `Error` | 47.53 | test_snapshots.py
   test_02_list_snapshots_with_removed_data_store | `Error` | 47.54 | test_snapshots.py
   ContextSuite context=TestSnapshotStandaloneBackup>:teardown | `Error` | 31.34 | test_snapshots.py
   test_01_snapshot_usage | `Error` | 25.61 | test_usage.py
   test_01_vpn_usage | `Error` | 1.08 | test_usage.py
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KVM: Enable HA heartbeat on ShareMountPoint [cloudstack]

Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #12773:
URL: https://github.com/apache/cloudstack/pull/12773#issuecomment-4136631933

   @weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org