You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by GitBox <gi...@apache.org> on 2021/02/18 20:26:27 UTC
[GitHub] [cloudstack] ustcweizhou opened a new pull request #4708: kvm: Handle storage issue on NFS/KVM in multiple ways
ustcweizhou opened a new pull request #4708:
URL: https://github.com/apache/cloudstack/pull/4708
### Description
This PR provides multiple options to handle storage issue on kvm.
FS: to be added
Two main parts
(1) kvm: Handle storage issue on NFS/KVM in multiple ways : hardreset (default), noaction, destroyvms, stopagent
(2) kvm: kvm: Schedule investigate tasks for disconnected hosts
<!--- Describe your changes in DETAIL - And how has behaviour functionally changed. -->
<!-- For new features, provide link to FS, dev ML discussion etc. -->
<!-- In case of bug fix, the expected and actual behaviours, steps to reproduce. -->
<!-- When "Fixes: #<id>" is specified, the issue/PR will automatically be closed when this PR gets merged -->
<!-- For addressing multiple issues/PRs, use multiple "Fixes: #<id>" -->
<!-- Fixes: # -->
<!--- ********************************************************************************* -->
<!--- NOTE: AUTOMATATION USES THE DESCRIPTIONS TO SET LABELS AND PRODUCE DOCUMENTATION. -->
<!--- PLEASE PUT AN 'X' in only **ONE** box -->
<!--- ********************************************************************************* -->
### Types of changes
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Bug fix (non-breaking change which fixes an issue)
- [X] Enhancement (improves an existing feature and functionality)
- [ ] Cleanup (Code refactoring and cleanup, that may add test cases)
### Feature/Enhancement Scale or Bug Severity
#### Feature/Enhancement Scale
- [ ] Major
- [X] Minor
#### Bug Severity
- [ ] BLOCKER
- [ ] Critical
- [ ] Major
- [ ] Minor
- [ ] Trivial
### Screenshots (if appropriate):
### How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
<!-- Please read the [CONTRIBUTING](https://github.com/apache/cloudstack/blob/master/CONTRIBUTING.md) document -->
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] weizhouapache closed pull request #4708: kvm: Handle storage issue on NFS/KVM in multiple ways
Posted by GitBox <gi...@apache.org>.
weizhouapache closed pull request #4708:
URL: https://github.com/apache/cloudstack/pull/4708
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] sureshanaparti commented on pull request #4708: kvm: Handle storage issue on NFS/KVM in multiple ways
Posted by GitBox <gi...@apache.org>.
sureshanaparti commented on pull request #4708:
URL: https://github.com/apache/cloudstack/pull/4708#issuecomment-995688938
> @weizhouapache I created a new pr for this in #5783
@weizhouapache @ravening is this PR no longer relevant (as the changes are ported to #5783)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] weizhouapache commented on a change in pull request #4708: kvm: Handle storage issue on NFS/KVM in multiple ways
Posted by GitBox <gi...@apache.org>.
weizhouapache commented on a change in pull request #4708:
URL: https://github.com/apache/cloudstack/pull/4708#discussion_r709942050
##########
File path: plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/KVMHAMonitor.java
##########
@@ -77,84 +92,148 @@ public NfsStoragePool getStoragePool(String uuid) {
}
}
- private class Monitor extends ManagedContextRunnable {
-
- @Override
- protected void runInContext() {
- synchronized (_storagePool) {
- Set<String> removedPools = new HashSet<String>();
- for (String uuid : _storagePool.keySet()) {
- NfsStoragePool primaryStoragePool = _storagePool.get(uuid);
+ protected class CheckPoolThread extends Thread {
Review comment:
ManagedContextRunnable and Thread are both implemetation of Runnable interface.
Thread works in testing.
ManagedContextRunnable should also be ok. The following might work as well
```
protected class CheckPoolThread implements Runnable {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] sureshanaparti commented on a change in pull request #4708: kvm: Handle storage issue on NFS/KVM in multiple ways
Posted by GitBox <gi...@apache.org>.
sureshanaparti commented on a change in pull request #4708:
URL: https://github.com/apache/cloudstack/pull/4708#discussion_r709886226
##########
File path: engine/orchestration/src/main/java/com/cloud/agent/manager/AgentManagerImpl.java
##########
@@ -203,6 +218,14 @@
protected final ConfigKey<Boolean> CheckTxnBeforeSending = new ConfigKey<Boolean>("Developer", Boolean.class, "check.txn.before.sending.agent.commands", "false",
"This parameter allows developers to enable a check to see if a transaction wraps commands that are sent to the resource. This is not to be enabled on production systems.", true);
+ protected final ConfigKey<Boolean> InvestigateDisconnectedHosts = new ConfigKey<>("Advanced", Boolean.class, "investigate.disconnected.hosts",
+ "false", "Determines whether to investigate VMs on disconnected hosts", false);
+ protected final ConfigKey<Integer> InvestigateDisconnectedHostsInterval = new ConfigKey<>("Advanced", Integer.class, "investigate.disconnected.hosts.interval",
+ "300", "The time (in seconds) between VM investigation on disconnected hosts.", false);
+ protected final ConfigKey<Integer> InvestigateDisconnectedHostsPoolSize = new ConfigKey<Integer>("Advanced", Integer.class, "investigate.disconnected.hosts.pool.size", "10",
+ "Default pool size to investigate disconnected hosts", false);
Review comment:
```suggestion
"The thread pool size to investigate disconnected hosts", false);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] weizhouapache commented on pull request #4708: kvm: Handle storage issue on NFS/KVM in multiple ways
Posted by GitBox <gi...@apache.org>.
weizhouapache commented on pull request #4708:
URL: https://github.com/apache/cloudstack/pull/4708#issuecomment-983523916
> @weizhouapache I hope you are doing well :) I'm going to work on this pr. How can I reproduce it?
@soreana
what do you want to reproduce ? this is an enhancement, not a bug.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] weizhouapache commented on pull request #4708: kvm: Handle storage issue on NFS/KVM in multiple ways
Posted by GitBox <gi...@apache.org>.
weizhouapache commented on pull request #4708:
URL: https://github.com/apache/cloudstack/pull/4708#issuecomment-983773720
> > > @weizhouapache I hope you are doing well :) I'm going to work on this pr. How can I reproduce it?
> >
> >
> > @soreana what do you want to reproduce ? this is an enhancement, not a bug.
>
> Some test cases to see if it works or not would be enough.
@soreana
I think you can get more info on your wiki/jira, etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] nvazquez commented on pull request #4708: kvm: Handle storage issue on NFS/KVM in multiple ways
Posted by GitBox <gi...@apache.org>.
nvazquez commented on pull request #4708:
URL: https://github.com/apache/cloudstack/pull/4708#issuecomment-866915598
Hi @ustcweizhou can you please resolve the conflicts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] soreana commented on pull request #4708: kvm: Handle storage issue on NFS/KVM in multiple ways
Posted by GitBox <gi...@apache.org>.
soreana commented on pull request #4708:
URL: https://github.com/apache/cloudstack/pull/4708#issuecomment-920738615
@weizhouapache I'm agree 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] DaanHoogland commented on a change in pull request #4708: kvm: Handle storage issue on NFS/KVM in multiple ways
Posted by GitBox <gi...@apache.org>.
DaanHoogland commented on a change in pull request #4708:
URL: https://github.com/apache/cloudstack/pull/4708#discussion_r661470982
##########
File path: plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/KVMHAMonitor.java
##########
@@ -77,84 +92,148 @@ public NfsStoragePool getStoragePool(String uuid) {
}
}
- private class Monitor extends ManagedContextRunnable {
-
- @Override
- protected void runInContext() {
- synchronized (_storagePool) {
- Set<String> removedPools = new HashSet<String>();
- for (String uuid : _storagePool.keySet()) {
- NfsStoragePool primaryStoragePool = _storagePool.get(uuid);
+ protected class CheckPoolThread extends Thread {
Review comment:
why not make this a ManagemedContextRunnable as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] ravening commented on pull request #4708: kvm: Handle storage issue on NFS/KVM in multiple ways
Posted by GitBox <gi...@apache.org>.
ravening commented on pull request #4708:
URL: https://github.com/apache/cloudstack/pull/4708#issuecomment-995586574
@weizhouapache I created a new pr for this in https://github.com/apache/cloudstack/pull/5783
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4708: kvm: Handle storage issue on NFS/KVM in multiple ways
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4708:
URL: https://github.com/apache/cloudstack/pull/4708#issuecomment-847929682
Packaging result: :heavy_check_mark: centos7 :heavy_check_mark: centos8 :heavy_check_mark: debian. SL-JID 91
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on pull request #4708: kvm: Handle storage issue on NFS/KVM in multiple ways
Posted by GitBox <gi...@apache.org>.
rhtyd commented on pull request #4708:
URL: https://github.com/apache/cloudstack/pull/4708#issuecomment-918844081
Ping @weizhouapache can you fix the conflicts, is this essential for 4.16?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] sureshanaparti commented on a change in pull request #4708: kvm: Handle storage issue on NFS/KVM in multiple ways
Posted by GitBox <gi...@apache.org>.
sureshanaparti commented on a change in pull request #4708:
URL: https://github.com/apache/cloudstack/pull/4708#discussion_r709883620
##########
File path: engine/orchestration/src/main/java/com/cloud/agent/manager/AgentManagerImpl.java
##########
@@ -238,6 +261,9 @@ public boolean configure(final String name, final Map<String, Object> params) th
_monitorExecutor = new ScheduledThreadPoolExecutor(1, new NamedThreadFactory("AgentMonitor"));
+ _scanHostsExecutor = new ScheduledThreadPoolExecutor(1, new NamedThreadFactory("HostsScanner"));
+ _investigatorExecutor = new ScheduledThreadPoolExecutor(InvestigateDisconnectedHostsPoolSize.value(), new NamedThreadFactory("DisconnectHostsInvestigator"));
Review comment:
```suggestion
_investigatorExecutor = new ScheduledThreadPoolExecutor(InvestigateDisconnectedHostsPoolSize.value(), new NamedThreadFactory("DisconnectedHostsInvestigator"));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] weizhouapache closed pull request #4708: kvm: Handle storage issue on NFS/KVM in multiple ways
Posted by GitBox <gi...@apache.org>.
weizhouapache closed pull request #4708:
URL: https://github.com/apache/cloudstack/pull/4708
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] soreana commented on pull request #4708: kvm: Handle storage issue on NFS/KVM in multiple ways
Posted by GitBox <gi...@apache.org>.
soreana commented on pull request #4708:
URL: https://github.com/apache/cloudstack/pull/4708#issuecomment-983547306
> > @weizhouapache I hope you are doing well :) I'm going to work on this pr. How can I reproduce it?
>
> @soreana what do you want to reproduce ? this is an enhancement, not a bug.
Some test cases to see if it works or not would be enough.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] weizhouapache commented on pull request #4708: kvm: Handle storage issue on NFS/KVM in multiple ways
Posted by GitBox <gi...@apache.org>.
weizhouapache commented on pull request #4708:
URL: https://github.com/apache/cloudstack/pull/4708#issuecomment-781631482
related to #2722 #2890 #2984 #4586
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] sureshanaparti commented on pull request #4708: kvm: Handle storage issue on NFS/KVM in multiple ways
Posted by GitBox <gi...@apache.org>.
sureshanaparti commented on pull request #4708:
URL: https://github.com/apache/cloudstack/pull/4708#issuecomment-920691042
@weizhouapache can you address outstanding remarks, and fix the conflicts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] weizhouapache commented on pull request #4708: kvm: Handle storage issue on NFS/KVM in multiple ways
Posted by GitBox <gi...@apache.org>.
weizhouapache commented on pull request #4708:
URL: https://github.com/apache/cloudstack/pull/4708#issuecomment-920721526
@rhtyd @sureshanaparti
this PR has many conflicts with #5239 which has been merged into master.
I think it is better to re-target to 4.17
what do you think ? @ravening @soreana
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] sureshanaparti commented on a change in pull request #4708: kvm: Handle storage issue on NFS/KVM in multiple ways
Posted by GitBox <gi...@apache.org>.
sureshanaparti commented on a change in pull request #4708:
URL: https://github.com/apache/cloudstack/pull/4708#discussion_r709884711
##########
File path: engine/orchestration/src/main/java/com/cloud/agent/manager/AgentManagerImpl.java
##########
@@ -773,6 +803,8 @@ public boolean stop() {
_connectExecutor.shutdownNow();
_monitorExecutor.shutdownNow();
+ _investigatorExecutor.shutdownNow();
+
Review comment:
`_scanHostsExecutor.shutdownNow()` missing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] soreana commented on pull request #4708: kvm: Handle storage issue on NFS/KVM in multiple ways
Posted by GitBox <gi...@apache.org>.
soreana commented on pull request #4708:
URL: https://github.com/apache/cloudstack/pull/4708#issuecomment-983515284
@weizhouapache I hope you are doing well :)
I'm going to work on this pr. How can I reproduce it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on pull request #4708: kvm: Handle storage issue on NFS/KVM in multiple ways
Posted by GitBox <gi...@apache.org>.
rhtyd commented on pull request #4708:
URL: https://github.com/apache/cloudstack/pull/4708#issuecomment-847893967
@blueorangutan package
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4708: kvm: Handle storage issue on NFS/KVM in multiple ways
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4708:
URL: https://github.com/apache/cloudstack/pull/4708#issuecomment-847894380
@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] GabrielBrascher commented on pull request #4708: kvm: Handle storage issue on NFS/KVM in multiple ways
Posted by GitBox <gi...@apache.org>.
GabrielBrascher commented on pull request #4708:
URL: https://github.com/apache/cloudstack/pull/4708#issuecomment-848184689
@ustcweizhou thanks for the PR. From what I understood (really quick glance of eyes) this avoid also situations of HA monitor a removed NFS storage. Am I right?
I have seen some weird issues where a Cluster that had NFS and removes it (standard storage removal process `Up > Maintenance > Removed`) but still HA expects NFS pool to be "Healthy" and simply Fence all hosts on the cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] ravening commented on pull request #4708: kvm: Handle storage issue on NFS/KVM in multiple ways
Posted by GitBox <gi...@apache.org>.
ravening commented on pull request #4708:
URL: https://github.com/apache/cloudstack/pull/4708#issuecomment-995690147
> > @weizhouapache I created a new pr for this in #5783
>
> @weizhouapache @ravening is this PR no longer relevant (as the changes are ported to #5783)?
@sureshanaparti yes. this has conflicts.. i have resolved the conflicts and created new pr
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org