You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by bhaisaab <gi...@git.apache.org> on 2016/04/20 10:16:16 UTC
[GitHub] cloudstack pull request: [WIP] Don't start review yet -- CLOUDSTAC...
GitHub user bhaisaab opened a pull request:
https://github.com/apache/cloudstack/pull/1502
[WIP] Don't start review yet -- CLOUDSTACK-9299: Out-of-band Management for CloudStack
Support access to a host’s out-of-band management interface (e.g. IPMI, iLO,
DRAC, etc.) to manage host power operations (on/off etc.) and querying current
power state in CloudStack.
Given the wide range of out-of-band management interfaces such as iLO and iDRA,
the service implementation allows for development of separate drivers as plugins.
This feature comes with a ipmitool based driver that uses the
ipmitool (http://linux.die.net/man/1/ipmitool) to communicate with any
out-of-band management interface that support IPMI 2.0.
This feature allows following common use-cases:
- Restarting stalled/failed hosts
- Powering off under-utilised hosts
- Powering on hosts for provisioning or to increase capacity
- Allowing system administrators to see the current power state of the host
For testing this feature `ipmisim` can be used:
https://pypi.python.org/pypi/ipmisim
FS:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Out-of-band+Management+for+CloudStack
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/shapeblue/cloudstack outofband-master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/cloudstack/pull/1502.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1502
----
commit 84157cebaa6bf3986efedb2887999d58653e8dd0
Author: Rohit Yadav <ro...@shapeblue.com>
Date: 2015-12-28T11:07:03Z
CLOUDSTACK-9299: Out-of-band Management for CloudStack
Support access to a host’s out-of-band management interface (e.g. IPMI, iLO,
DRAC, etc.) to manage host power operations (on/off etc.) and querying current
power state in CloudStack.
Given the wide range of out-of-band management interfaces such as iLO and iDRA,
the service implementation allows for development of separate drivers as plugins.
This feature comes with a ipmitool based driver that uses the
ipmitool (http://linux.die.net/man/1/ipmitool) to communicate with any
out-of-band management interface that support IPMI 2.0.
This feature allows following common use-cases:
- Restarting stalled/failed hosts
- Powering off under-utilised hosts
- Powering on hosts for provisioning or to increase capacity
- Allowing system administrators to see the current power state of the host
For testing this feature `ipmisim` can be used:
https://pypi.python.org/pypi/ipmisim
FS:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Out-of-band+Management+for+CloudStack
Signed-off-by: Rohit Yadav <ro...@shapeblue.com>
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-214636312
Thank you @DaanHoogland @pyr @wido for the review
@swill I think we've enough votes for this one, though le'ts wait for @borisstoyanov 's test results which he has told me he'll share soon
@jburwell do you want to review and suggest any changes? thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61845867
--- Diff: plugins/outofbandmanagement-drivers/ipmitool/src/org/apache/cloudstack/outofbandmanagement/driver/ipmitool/IpmitoolWrapper.java ---
@@ -0,0 +1,166 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.outofbandmanagement.driver.ipmitool;
+
+import com.cloud.utils.StringUtils;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.google.common.base.Strings;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagement;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverResponse;
+import org.apache.cloudstack.utils.process.ProcessRunner;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.List;
+
+public class IpmitoolWrapper {
+ public static final Logger LOG = Logger.getLogger(IpmitoolWrapper.class);
+
+ public static String parsePowerCommand(OutOfBandManagement.PowerOperation operation) {
+ if (operation == null) {
+ throw new IllegalStateException("Invalid power operation requested");
+ }
+ switch (operation) {
+ case ON:
+ case OFF:
+ case CYCLE:
+ case RESET:
+ case SOFT:
+ case STATUS:
+ break;
+ default:
+ throw new IllegalStateException("Invalid power operation requested");
+ }
+ return operation.toString().toLowerCase();
+ }
+
+ public static OutOfBandManagement.PowerState parsePowerState(final String standardOutput) {
+ if (Strings.isNullOrEmpty(standardOutput)) {
+ return OutOfBandManagement.PowerState.Unknown;
+ }
+ if (standardOutput.equals("Chassis Power is on")) {
+ return OutOfBandManagement.PowerState.On;
+ } else if (standardOutput.equals("Chassis Power is off")) {
+ return OutOfBandManagement.PowerState.Off;
+ }
+ return OutOfBandManagement.PowerState.Unknown;
+ }
+
+ public static List<String> getIpmiToolCommandArgs(final String ipmiToolPath, final String ipmiInterface, final String retries,
+ final ImmutableMap<OutOfBandManagement.Option, String> options, String... commands) {
+
+ ImmutableList.Builder<String> ipmiToolCommands = ImmutableList.<String>builder()
+ .add(ipmiToolPath)
+ .add("-I")
+ .add(ipmiInterface)
+ .add("-R")
+ .add(retries)
+ .add("-v");
+
+ if (options != null) {
+ for (ImmutableMap.Entry<OutOfBandManagement.Option, String> option : options.entrySet()) {
+ switch (option.getKey()) {
+ case ADDRESS:
+ ipmiToolCommands.add("-H");
+ break;
+ case PORT:
+ ipmiToolCommands.add("-p");
+ break;
+ case USERNAME:
+ ipmiToolCommands.add("-U");
+ break;
+ case PASSWORD:
+ ipmiToolCommands.add("-P");
+ break;
+ default:
+ continue;
+ }
+ ipmiToolCommands.add(option.getValue());
+ }
+ }
+ for (String command : commands) {
+ ipmiToolCommands.add(command);
+ }
+ return ipmiToolCommands.build();
+ }
+
+ public static String findIpmiUser(final String usersList, final String username) {
+ // Expected usersList string contains legends on first line and users on rest
+ // ID Name Callin Link Auth IPMI Msg Channel Priv Limit
+ // 1 admin true true true ADMINISTRATOR
--- End diff --
Fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61847113
--- Diff: server/src/org/apache/cloudstack/outofbandmanagement/OutOfBandManagementServiceImpl.java ---
@@ -0,0 +1,532 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.outofbandmanagement;
+
+import com.cloud.alert.AlertManager;
+import com.cloud.dc.ClusterDetailsDao;
+import com.cloud.dc.ClusterDetailsVO;
+import com.cloud.dc.DataCenter;
+import com.cloud.dc.DataCenterDetailVO;
+import com.cloud.dc.dao.DataCenterDetailsDao;
+import com.cloud.domain.Domain;
+import com.cloud.event.ActionEvent;
+import com.cloud.event.ActionEventUtils;
+import com.cloud.event.EventTypes;
+import com.cloud.host.Host;
+import com.cloud.host.dao.HostDao;
+import com.cloud.org.Cluster;
+import com.cloud.utils.component.Manager;
+import com.cloud.utils.component.ManagerBase;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.db.Transaction;
+import com.cloud.utils.db.TransactionCallback;
+import com.cloud.utils.db.TransactionStatus;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.fsm.NoTransitionException;
+import com.google.common.base.Strings;
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import com.google.common.collect.ImmutableMap;
+import org.apache.cloudstack.api.response.OutOfBandManagementResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.framework.config.ConfigKey;
+import org.apache.cloudstack.framework.config.Configurable;
+import org.apache.cloudstack.outofbandmanagement.dao.OutOfBandManagementDao;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverChangePasswordCommand;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverPowerCommand;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverResponse;
+import org.apache.cloudstack.utils.identity.ManagementServerNode;
+import org.apache.log4j.Logger;
+import org.springframework.stereotype.Component;
+
+import javax.ejb.Local;
+import javax.inject.Inject;
+import javax.naming.ConfigurationException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+@Component
+@Local(value = {OutOfBandManagementService.class})
+public class OutOfBandManagementServiceImpl extends ManagerBase implements OutOfBandManagementService, Manager, Configurable {
+ public static final Logger LOG = Logger.getLogger(OutOfBandManagementServiceImpl.class);
+
+ @Inject
+ private ClusterDetailsDao clusterDetailsDao;
+ @Inject
+ private DataCenterDetailsDao dataCenterDetailsDao;
+ @Inject
+ private OutOfBandManagementDao outOfBandManagementDao;
+ @Inject
+ private HostDao hostDao;
+ @Inject
+ private AlertManager alertMgr;
+
+ private String name;
+ private long serviceId;
+
+ private List<OutOfBandManagementDriver> outOfBandManagementDrivers = new ArrayList<>();
+ private Map<String, OutOfBandManagementDriver> outOfBandManagementDriversMap = new HashMap<String, OutOfBandManagementDriver>();
+
+ private static final String OOBM_ENABLED_DETAIL = "outOfBandManagementEnabled";
+ private static final int ACQUIRE_GLOBAL_LOCK_TIMEOUT_FOR_HOST = 120;
+
+ private Cache<Long, Long> hostAlertCache;
+ private static ExecutorService backgroundSyncExecutor;
+
+ private String getOutOfBandManagementHostLock(long id) {
+ return "oobm.host." + id;
+ }
+
+ private void initializeDriversMap() {
+ if (outOfBandManagementDriversMap.isEmpty() && outOfBandManagementDrivers != null && outOfBandManagementDrivers.size() > 0) {
+ for (final OutOfBandManagementDriver driver : outOfBandManagementDrivers) {
+ outOfBandManagementDriversMap.put(driver.getName().toLowerCase(), driver);
+ }
+ LOG.debug("Discovered out-of-band management drivers configured in the OutOfBandManagementService");
+ }
+ }
+
+ private OutOfBandManagementDriver getDriver(final OutOfBandManagement outOfBandManagementConfig) {
+ if (!Strings.isNullOrEmpty(outOfBandManagementConfig.getDriver())) {
+ final OutOfBandManagementDriver driver = outOfBandManagementDriversMap.get(outOfBandManagementConfig.getDriver());
+ if (driver != null) {
+ return driver;
+ }
+ }
+ throw new CloudRuntimeException("Configured out-of-band management driver is not available. Aborting any out-of-band management action.");
+ }
+
+ protected OutOfBandManagement updateConfig(final OutOfBandManagement outOfBandManagementConfig, final ImmutableMap<OutOfBandManagement.Option, String> options) {
+ if (outOfBandManagementConfig == null) {
+ throw new CloudRuntimeException("Out-of-band management is not configured for the host. Aborting.");
+ }
+ if (options == null) {
+ return outOfBandManagementConfig;
+ }
+ for (OutOfBandManagement.Option option: options.keySet()) {
+ final String value = options.get(option);
+ if (Strings.isNullOrEmpty(value)) {
+ continue;
+ }
+ switch (option) {
+ case DRIVER:
+ outOfBandManagementConfig.setDriver(value);
+ break;
+ case ADDRESS:
+ outOfBandManagementConfig.setAddress(value);
+ break;
+ case PORT:
+ outOfBandManagementConfig.setPort(Integer.parseInt(value));
+ break;
+ case USERNAME:
+ outOfBandManagementConfig.setUsername(value);
+ break;
+ case PASSWORD:
+ outOfBandManagementConfig.setPassword(value);
+ break;
+ }
+ }
+ return outOfBandManagementConfig;
+ }
+
+ protected ImmutableMap<OutOfBandManagement.Option, String> getOptions(final OutOfBandManagement outOfBandManagementConfig) {
+ final ImmutableMap.Builder<OutOfBandManagement.Option, String> optionsBuilder = ImmutableMap.builder();
+ if (outOfBandManagementConfig == null) {
+ throw new CloudRuntimeException("Out-of-band management is not configured for the host. Aborting.");
+ }
+ for (OutOfBandManagement.Option option: OutOfBandManagement.Option.values()) {
+ String value = null;
+ switch (option) {
+ case DRIVER:
+ value = outOfBandManagementConfig.getDriver();
+ break;
+ case ADDRESS:
+ value = outOfBandManagementConfig.getAddress();
+ break;
+ case PORT:
+ if (outOfBandManagementConfig.getPort() != null) {
+ value = String.valueOf(outOfBandManagementConfig.getPort());
+ }
+ break;
+ case USERNAME:
+ value = outOfBandManagementConfig.getUsername();
+ break;
+ case PASSWORD:
+ value = outOfBandManagementConfig.getPassword();
+ break;
+ }
+ if (value != null) {
+ optionsBuilder.put(option, value);
+ }
+ }
+ return optionsBuilder.build();
+ }
+
+ private void sendAuthError(final Host host, final String message) {
+ try {
+ hostAlertCache.asMap().putIfAbsent(host.getId(), 0L);
+ Long sentCount = hostAlertCache.asMap().get(host.getId());
+ if (sentCount != null && sentCount <= 0) {
+ boolean concurrentUpdateResult = hostAlertCache.asMap().replace(host.getId(), sentCount, sentCount+1L);
+ if (concurrentUpdateResult) {
+ final String subject = String.format("Out-of-band management auth-error detected for host:%d in cluster:%d, zone:%d", host.getId(), host.getClusterId(), host.getDataCenterId());
+ LOG.error(subject + ": " + message);
+ alertMgr.sendAlert(AlertManager.AlertType.ALERT_TYPE_OOBM_AUTH_ERROR, host.getDataCenterId(), host.getPodId(), subject, message);
+ }
+ }
+ } catch (Exception ignored) {
+ }
+ }
+
+ private boolean transitionPowerState(OutOfBandManagement.PowerState.Event event, OutOfBandManagement outOfBandManagementHost) {
+ if (outOfBandManagementHost == null) {
+ return false;
+ }
+ OutOfBandManagement.PowerState currentPowerState = outOfBandManagementHost.getPowerState();
+ try {
+ OutOfBandManagement.PowerState newPowerState = OutOfBandManagement.PowerState.getStateMachine().getNextState(currentPowerState, event);
+ boolean result = outOfBandManagementDao.updateState(currentPowerState, event, newPowerState, outOfBandManagementHost, null);
+ if (result) {
+ final String message = String.format("Transitioned out-of-band management power state from:%s to:%s due to event:%s for the host id:%d", currentPowerState, newPowerState, event, outOfBandManagementHost.getHostId());
+ LOG.debug(message);
+ ActionEventUtils.onActionEvent(CallContext.current().getCallingUserId(), CallContext.current().getCallingAccountId(), Domain.ROOT_DOMAIN,
+ EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_POWERSTATE_TRANSITION, message);
+ }
+ return result;
+ } catch (NoTransitionException ignored) {
+ LOG.trace(String.format("Unable to transition out-of-band management power state for host id=%s for the event=%s and current power state=%s", outOfBandManagementHost.getHostId(), event, currentPowerState));
+ }
+ return false;
+ }
+
+ private boolean isOutOfBandManagementEnabledForZone(Long zoneId) {
+ if (zoneId == null) {
+ return true;
+ }
+ final DataCenterDetailVO zoneDetails = dataCenterDetailsDao.findDetail(zoneId, OOBM_ENABLED_DETAIL);
+ if (zoneDetails != null && !Strings.isNullOrEmpty(zoneDetails.getValue()) && !Boolean.valueOf(zoneDetails.getValue())) {
+ return false;
+ }
+ return true;
+ }
+
+ private boolean isOutOfBandManagementEnabledForCluster(Long clusterId) {
+ if (clusterId == null) {
+ return true;
+ }
+ final ClusterDetailsVO clusterDetails = clusterDetailsDao.findDetail(clusterId, OOBM_ENABLED_DETAIL);
+ if (clusterDetails != null && !Strings.isNullOrEmpty(clusterDetails.getValue()) && !Boolean.valueOf(clusterDetails.getValue())) {
+ return false;
+ }
+ return true;
+ }
+
+ private boolean isOutOfBandManagementEnabledForHost(Long hostId) {
+ if (hostId == null) {
+ return false;
+ }
+ final OutOfBandManagement outOfBandManagementConfig = outOfBandManagementDao.findByHost(hostId);
+ if (outOfBandManagementConfig == null || !outOfBandManagementConfig.isEnabled()) {
+ return false;
+ }
+ return true;
+ }
+
+ private void checkOutOfBandManagementEnabledByZoneClusterHost(final Host host) {
+ if (!isOutOfBandManagementEnabledForZone(host.getDataCenterId())) {
+ throw new CloudRuntimeException("Out-of-band management is disabled for the host's zone. Aborting Operation.");
+ }
+ if (!isOutOfBandManagementEnabledForCluster(host.getClusterId())) {
+ throw new CloudRuntimeException("Out-of-band management is disabled for the host's cluster. Aborting Operation.");
+ }
+ if (!isOutOfBandManagementEnabledForHost(host.getId())) {
+ throw new CloudRuntimeException("Out-of-band management is disabled or not configured for the host. Aborting Operation.");
+ }
+ }
+
+ public boolean isOutOfBandManagementEnabled(final Host host) {
+ return isOutOfBandManagementEnabledForZone(host.getDataCenterId())
+ && isOutOfBandManagementEnabledForCluster(host.getClusterId())
+ && isOutOfBandManagementEnabledForHost(host.getId());
+ }
+
+ public boolean transitionPowerStateToDisabled(List<? extends Host> hosts) {
+ boolean result = true;
+ for (Host host : hosts) {
+ result = result && transitionPowerState(OutOfBandManagement.PowerState.Event.Disabled,
+ outOfBandManagementDao.findByHost(host.getId()));
+ }
+ return result;
+ }
+
+ public void submitBackgroundPowerSyncTask(final Host host) {
+ if (host != null) {
+ // Note: This is a blocking queue based executor
+ backgroundSyncExecutor.submit(new OutOfBandManagementBackgroundTask(this, host, OutOfBandManagement.PowerOperation.STATUS));
+ }
+ }
+
+ private OutOfBandManagementResponse buildEnableDisableResponse(final boolean enabled) {
+ final OutOfBandManagementResponse response = new OutOfBandManagementResponse();
+ response.setEnabled(enabled);
+ response.setSuccess(true);
+ return response;
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_ENABLEDISABLE, eventDescription = "enabling/disabling out-of-band management on a zone")
+ public OutOfBandManagementResponse enableDisableOutOfBandManagement(final DataCenter zone, final boolean enabled) {
+ dataCenterDetailsDao.persist(zone.getId(), OOBM_ENABLED_DETAIL, String.valueOf(enabled));
+ if (!enabled) {
+ transitionPowerStateToDisabled(hostDao.findByDataCenterId(zone.getId()));
+ }
+ return buildEnableDisableResponse(enabled);
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_ENABLEDISABLE, eventDescription = "enabling/disabling out-of-band management on a cluster")
+ public OutOfBandManagementResponse enableDisableOutOfBandManagement(final Cluster cluster, final boolean enabled) {
+ clusterDetailsDao.persist(cluster.getId(), OOBM_ENABLED_DETAIL, String.valueOf(enabled));
+ if (!enabled) {
+ transitionPowerStateToDisabled(hostDao.findByClusterId(cluster.getId()));
+ }
+ return buildEnableDisableResponse(enabled);
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_ENABLEDISABLE, eventDescription = "enabling/disabling out-of-band management on a host")
+ public OutOfBandManagementResponse enableDisableOutOfBandManagement(final Host host, final boolean enabled) {
+ final OutOfBandManagement outOfBandManagementConfig = outOfBandManagementDao.findByHost(host.getId());
+ if (outOfBandManagementConfig == null) {
+ final OutOfBandManagementResponse response = new OutOfBandManagementResponse(null);
+ response.setSuccess(false);
+ response.setResultDescription("Out-of-band management is not configured for the host. Please configure the host before enabling/disabling it.");
+ return response;
+ }
+ hostAlertCache.invalidate(host.getId());
+ outOfBandManagementConfig.setEnabled(enabled);
+ boolean updateResult = outOfBandManagementDao.update(outOfBandManagementConfig.getId(), (OutOfBandManagementVO) outOfBandManagementConfig);
+ if (updateResult && !enabled) {
+ transitionPowerStateToDisabled(Collections.singletonList(host));
+ }
+ return buildEnableDisableResponse(enabled && updateResult);
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_CONFIGURE, eventDescription = "updating out-of-band management configuration")
+ public OutOfBandManagementResponse configureOutOfBandManagement(final Host host, final ImmutableMap<OutOfBandManagement.Option, String> options) {
+ OutOfBandManagement outOfBandManagementConfig = outOfBandManagementDao.findByHost(host.getId());
+ if (outOfBandManagementConfig == null) {
+ outOfBandManagementConfig = outOfBandManagementDao.persist(new OutOfBandManagementVO(host.getId()));
+ }
+ outOfBandManagementConfig = updateConfig(outOfBandManagementConfig, options);
+ if (Strings.isNullOrEmpty(outOfBandManagementConfig.getDriver()) || !outOfBandManagementDriversMap.containsKey(outOfBandManagementConfig.getDriver().toLowerCase())) {
+ throw new CloudRuntimeException("Out-of-band management driver is not available. Please provide a valid driver name.");
+ }
+
+ boolean updatedConfig = outOfBandManagementDao.update(outOfBandManagementConfig.getId(), (OutOfBandManagementVO) outOfBandManagementConfig);
+ CallContext.current().setEventDetails("host id:" + host.getId() + " configuration:" + outOfBandManagementConfig.getAddress() + ":" + outOfBandManagementConfig.getPort());
+
+ if (!updatedConfig) {
+ throw new CloudRuntimeException("Failed to update out-of-band management config for the host in the database.");
+ }
+
+ String result = "Out-of-band management successfully configured for the host";
+ LOG.debug(result);
+
+ final OutOfBandManagementResponse response = new OutOfBandManagementResponse(outOfBandManagementDao.findByHost(host.getId()));
+ response.setResultDescription(result);
+ response.setSuccess(true);
+ return response;
--- End diff --
that method is only for enable/disable APIs. Also, this code accepts the config to populate other fields; while enable/disable response helper only cares about enable/disable operation status (true/false) and not the result description or other fields
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by nvazquez <gi...@git.apache.org>.
Github user nvazquez commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-218520519
@rhtyd @kiwiflyer @swill a PR for fixing the problem #1539
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61845251
--- Diff: engine/schema/src/org/apache/cloudstack/outofbandmanagement/dao/OutOfBandManagementDaoImpl.java ---
@@ -0,0 +1,163 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.outofbandmanagement.dao;
+
+import com.cloud.utils.DateUtil;
+import com.cloud.utils.db.Attribute;
+import com.cloud.utils.db.DB;
+import com.cloud.utils.db.Filter;
+import com.cloud.utils.db.GenericDaoBase;
+import com.cloud.utils.db.SearchBuilder;
+import com.cloud.utils.db.SearchCriteria;
+import com.cloud.utils.db.TransactionLegacy;
+import com.cloud.utils.db.UpdateBuilder;
+import com.cloud.utils.exception.CloudRuntimeException;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagement;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagementVO;
+import org.apache.log4j.Logger;
+import org.springframework.stereotype.Component;
+
+import javax.ejb.Local;
+import java.sql.PreparedStatement;
+import java.sql.SQLException;
+import java.util.List;
+
+@DB
+@Component
+@Local(value = {OutOfBandManagementDao.class})
+public class OutOfBandManagementDaoImpl extends GenericDaoBase<OutOfBandManagementVO, Long> implements OutOfBandManagementDao {
+ private static final Logger LOG = Logger.getLogger(OutOfBandManagementDaoImpl.class);
+
+ private SearchBuilder<OutOfBandManagementVO> HostSearch;
+ private SearchBuilder<OutOfBandManagementVO> ManagementServerSearch;
+ private SearchBuilder<OutOfBandManagementVO> OutOfBandManagementOwnerSearch;
+ private SearchBuilder<OutOfBandManagementVO> StateUpdateSearch;
+
+ private Attribute PowerStateAttr;
+ private Attribute MsIdAttr;
+ private Attribute UpdateTimeAttr;
+
+ public OutOfBandManagementDaoImpl() {
+ super();
+
+ HostSearch = createSearchBuilder();
+ HostSearch.and("hostId", HostSearch.entity().getHostId(), SearchCriteria.Op.EQ);
+ HostSearch.done();
+
+ ManagementServerSearch = createSearchBuilder();
+ ManagementServerSearch.and("server", ManagementServerSearch.entity().getManagementServerId(), SearchCriteria.Op.EQ);
+ ManagementServerSearch.done();
+
+ OutOfBandManagementOwnerSearch = createSearchBuilder();
+ OutOfBandManagementOwnerSearch.and("server", OutOfBandManagementOwnerSearch.entity().getManagementServerId(), SearchCriteria.Op.EQ);
+ OutOfBandManagementOwnerSearch.or("serverNull", OutOfBandManagementOwnerSearch.entity().getManagementServerId(), SearchCriteria.Op.NULL);
+ OutOfBandManagementOwnerSearch.done();
+
+ StateUpdateSearch = createSearchBuilder();
+ StateUpdateSearch.and("status", StateUpdateSearch.entity().getPowerState(), SearchCriteria.Op.EQ);
+ StateUpdateSearch.and("id", StateUpdateSearch.entity().getId(), SearchCriteria.Op.EQ);
+ StateUpdateSearch.and("update", StateUpdateSearch.entity().getUpdateCount(), SearchCriteria.Op.EQ);
+ StateUpdateSearch.done();
+
+ PowerStateAttr = _allAttributes.get("powerState");
+ MsIdAttr = _allAttributes.get("managementServerId");
+ UpdateTimeAttr = _allAttributes.get("updateTime");
+ assert (PowerStateAttr != null && MsIdAttr != null && UpdateTimeAttr != null) : "Couldn't find one of these attributes";
+ }
+
+ @Override
+ public OutOfBandManagement findByHost(long hostId) {
+ SearchCriteria<OutOfBandManagementVO> sc = HostSearch.create("hostId", hostId);
+ return findOneBy(sc);
+ }
+
+ @Override
+ public List<OutOfBandManagementVO> findAllByManagementServer(long serverId) {
+ SearchCriteria<OutOfBandManagementVO> sc = OutOfBandManagementOwnerSearch.create();
+ sc.setParameters("server", serverId);
+ return listBy(sc, new Filter(OutOfBandManagementVO.class, "updateTime", true, null, null));
+ }
+
+ private void executeExpireOwnershipSql(final String sql, long resource) {
+ TransactionLegacy txn = TransactionLegacy.currentTxn();
+ try {
+ txn.start();
+ PreparedStatement pstmt = txn.prepareAutoCloseStatement(sql);
+ pstmt.setLong(1, resource);
+ pstmt.executeUpdate();
+ txn.commit();
+ } catch (SQLException e) {
+ txn.rollback();
+ throw new CloudRuntimeException("Unable to reset out-of-band management ownership based on resource:" + resource);
+ }
+ }
+
+ @Override
+ public void expireOutOfBandManagementOwnershipByHours(long hours) {
+ final String resetOwnerSql = "UPDATE oobm set mgmt_server_id=NULL where update_time<= (NOW() - INTERVAL ? HOUR)";
+ executeExpireOwnershipSql(resetOwnerSql, hours);
+ }
+
+ @Override
+ public void expireOutOfBandManagementOwnershipByServer(long serverId) {
+ final String resetOwnerSql = "UPDATE oobm set mgmt_server_id=NULL, power_state=NULL where mgmt_server_id=?";
+ executeExpireOwnershipSql(resetOwnerSql, serverId);
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Expired out-of-band management ownership for hosts owned by management server id:" + serverId);
+ }
+ }
+
+ @Override
+ public boolean updateState(OutOfBandManagement.PowerState oldStatus, OutOfBandManagement.PowerState.Event event, OutOfBandManagement.PowerState newStatus, OutOfBandManagement vo, Object data) {
+ // lock target row from beginning to avoid lock-promotion caused deadlock
+ OutOfBandManagementVO oobmHost = lockRow(vo.getId(), true);
+ if (oobmHost == null) {
+ if (LOG.isTraceEnabled()) {
+ LOG.trace("Failed to lock row to update state for out-of-band management host id: " + vo.getHostId());
+ }
+ return false;
+ }
+
+ Long newManagementServerId = event.getServerId();
+ // Avoid updates when old ownership and state are same as new
+ if (oldStatus == newStatus && (oobmHost.getManagementServerId() != null && oobmHost.getManagementServerId().equals(newManagementServerId))) {
+ return false;
+ }
+
+ if (event == OutOfBandManagement.PowerState.Event.Disabled) {
+ newManagementServerId = null;
+ }
+
+ SearchCriteria<OutOfBandManagementVO> sc = StateUpdateSearch.create();
+ sc.setParameters("status", oldStatus);
+ sc.setParameters("id", oobmHost.getId());
+ sc.setParameters("update", oobmHost.getUpdateCount());
+
+ oobmHost.incrUpdateCount();
+ UpdateBuilder ub = getUpdateBuilder(oobmHost);
+ ub.set(oobmHost, PowerStateAttr, newStatus);
+ ub.set(oobmHost, UpdateTimeAttr, DateUtil.currentGMTTime());
+ ub.set(oobmHost, MsIdAttr, newManagementServerId);
+
+ int result = update(ub, sc, null);
+ if (LOG.isDebugEnabled() && result <= 0) {
+ LOG.debug(String.format("Failed to update out-of-band management power state from:%s to:%s due to event:%s for the host id:%d", oldStatus, newStatus, event, oobmHost.getHostId()));
--- End diff --
It is triggered by background sync thread, I've avoided to use WARN to spam the logs file in case of large prod. In case of frequent updates this can cause a lot of log entries, and when states are same.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61804388
--- Diff: server/src/org/apache/cloudstack/outofbandmanagement/OutOfBandManagementServiceImpl.java ---
@@ -0,0 +1,532 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.outofbandmanagement;
+
+import com.cloud.alert.AlertManager;
+import com.cloud.dc.ClusterDetailsDao;
+import com.cloud.dc.ClusterDetailsVO;
+import com.cloud.dc.DataCenter;
+import com.cloud.dc.DataCenterDetailVO;
+import com.cloud.dc.dao.DataCenterDetailsDao;
+import com.cloud.domain.Domain;
+import com.cloud.event.ActionEvent;
+import com.cloud.event.ActionEventUtils;
+import com.cloud.event.EventTypes;
+import com.cloud.host.Host;
+import com.cloud.host.dao.HostDao;
+import com.cloud.org.Cluster;
+import com.cloud.utils.component.Manager;
+import com.cloud.utils.component.ManagerBase;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.db.Transaction;
+import com.cloud.utils.db.TransactionCallback;
+import com.cloud.utils.db.TransactionStatus;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.fsm.NoTransitionException;
+import com.google.common.base.Strings;
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import com.google.common.collect.ImmutableMap;
+import org.apache.cloudstack.api.response.OutOfBandManagementResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.framework.config.ConfigKey;
+import org.apache.cloudstack.framework.config.Configurable;
+import org.apache.cloudstack.outofbandmanagement.dao.OutOfBandManagementDao;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverChangePasswordCommand;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverPowerCommand;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverResponse;
+import org.apache.cloudstack.utils.identity.ManagementServerNode;
+import org.apache.log4j.Logger;
+import org.springframework.stereotype.Component;
+
+import javax.ejb.Local;
+import javax.inject.Inject;
+import javax.naming.ConfigurationException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+@Component
+@Local(value = {OutOfBandManagementService.class})
+public class OutOfBandManagementServiceImpl extends ManagerBase implements OutOfBandManagementService, Manager, Configurable {
+ public static final Logger LOG = Logger.getLogger(OutOfBandManagementServiceImpl.class);
+
+ @Inject
+ private ClusterDetailsDao clusterDetailsDao;
+ @Inject
+ private DataCenterDetailsDao dataCenterDetailsDao;
+ @Inject
+ private OutOfBandManagementDao outOfBandManagementDao;
+ @Inject
+ private HostDao hostDao;
+ @Inject
+ private AlertManager alertMgr;
+
+ private String name;
+ private long serviceId;
+
+ private List<OutOfBandManagementDriver> outOfBandManagementDrivers = new ArrayList<>();
+ private Map<String, OutOfBandManagementDriver> outOfBandManagementDriversMap = new HashMap<String, OutOfBandManagementDriver>();
+
+ private static final String OOBM_ENABLED_DETAIL = "outOfBandManagementEnabled";
+ private static final int ACQUIRE_GLOBAL_LOCK_TIMEOUT_FOR_HOST = 120;
+
+ private Cache<Long, Long> hostAlertCache;
+ private static ExecutorService backgroundSyncExecutor;
--- End diff --
Why is this field declared as a non-final static?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61848360
--- Diff: test/integration/smoke/test_outofbandmanagement.py ---
@@ -0,0 +1,561 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import marvin
+from marvin.cloudstackTestCase import *
+from marvin.cloudstackAPI import *
+from marvin.lib.utils import *
+from marvin.lib.base import *
+from marvin.lib.common import *
+from marvin.lib.utils import (random_gen)
+from nose.plugins.attrib import attr
+
+from ipmisim.ipmisim import IpmiServerContext, IpmiServer, ThreadedIpmiServer
+
+import socket
+import sys
+import thread
+import time
+
+
+class TestOutOfBandManagement(cloudstackTestCase):
+ """ Test cases for out of band management
+ """
+
+ def setUp(self):
+ self.apiclient = self.testClient.getApiClient()
+ self.hypervisor = self.testClient.getHypervisorInfo()
+ self.dbclient = self.testClient.getDbConnection()
+ self.services = self.testClient.getParsedTestDataConfig()
+ self.mgtSvrDetails = self.config.__dict__["mgtSvr"][0].__dict__
+
+ self.zone = get_zone(self.apiclient, self.testClient.getZoneForTests())
+ self.host = None
+ self.server = None
+
+ # use random port for ipmisim
+ s = socket.socket()
+ s.bind(('', 0))
+ self.serverPort = s.getsockname()[1]
+ s.close()
+
+ self.cleanup = []
+
+
+ def tearDown(self):
+ try:
+ self.dbclient.execute("delete from oobm where port=%d" % self.getIpmiServerPort())
+ self.dbclient.execute("delete from mshost_peer where peer_runid=%s" % self.getFakeMsRunId())
+ self.dbclient.execute("delete from mshost where runid=%s" % self.getFakeMsRunId())
+ self.dbclient.execute("delete from cluster_details where name='outOfBandManagementEnabled'")
+ self.dbclient.execute("delete from data_center_details where name='outOfBandManagementEnabled'")
+ cleanup_resources(self.apiclient, self.cleanup)
+ if self.server:
+ self.server.shutdown()
+ self.server.server_close()
+ except Exception as e:
+ raise Exception("Warning: Exception during cleanup : %s" % e)
+
+
+ def getFakeMsId(self):
+ return 1234567890
+
+
+ def getFakeMsRunId(self):
+ return 123456
+
+
+ def getHost(self, hostId=None):
+ if self.host and hostId is None:
+ return self.host
+
+ response = list_hosts(
+ self.apiclient,
+ zoneid=self.zone.id,
+ type='Routing',
+ id=hostId
+ )
+ if len(response) > 0:
+ self.host = response[0]
+ return self.host
+ raise self.skipTest("No hosts found, skipping out-of-band management test")
+
+
+ def getIpmiServerIp(self):
+ s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
+ s.connect((self.mgtSvrDetails["mgtSvrIp"], self.mgtSvrDetails["port"]))
+ return s.getsockname()[0]
+
+
+ def getIpmiServerPort(self):
+ return self.serverPort
+
+
+ def getOobmConfigCmd(self):
+ cmd = configureOutOfBandManagement.configureOutOfBandManagementCmd()
+ cmd.driver = 'ipmitool' # The default available driver
+ cmd.address = self.getIpmiServerIp()
+ cmd.port = self.getIpmiServerPort()
+ cmd.username = 'admin'
+ cmd.password = 'password'
+ cmd.hostid = self.getHost().id
+ return cmd
+
+
+ def getOobmEnableCmd(self):
+ cmd = enableOutOfBandManagementForHost.enableOutOfBandManagementForHostCmd()
+ cmd.hostid = self.getHost().id
+ return cmd
+
+
+ def getOobmDisableCmd(self):
+ cmd = disableOutOfBandManagementForHost.disableOutOfBandManagementForHostCmd()
+ cmd.hostid = self.getHost().id
+ return cmd
+
+
+ def getOobmIssueActionCmd(self):
+ cmd = issueOutOfBandManagementPowerAction.issueOutOfBandManagementPowerActionCmd()
+ cmd.hostid = self.getHost().id
+ cmd.action = 'STATUS'
+ return cmd
+
+
+ def issuePowerActionCmd(self, action, timeout=None):
+ cmd = self.getOobmIssueActionCmd()
+ cmd.action = action
+ if timeout:
+ cmd.timeout = timeout
+ return self.apiclient.issueOutOfBandManagementPowerAction(cmd)
+
+
+ def configureAndEnableOobm(self):
+ self.apiclient.configureOutOfBandManagement(self.getOobmConfigCmd())
+ response = self.apiclient.enableOutOfBandManagementForHost(self.getOobmEnableCmd())
+ self.assertEqual(response.enabled, True)
+
+
+ def startIpmiServer(self):
+ def startIpmiServer(tname, server):
+ self.debug("Starting ipmisim server")
+ try:
+ server.serve_forever()
+ except Exception: pass
+ IpmiServerContext('reset')
+ ThreadedIpmiServer.allow_reuse_address = False
+ server = ThreadedIpmiServer(('0.0.0.0', self.getIpmiServerPort()), IpmiServer)
+ thread.start_new_thread(startIpmiServer, ("ipmi-server", server,))
+ self.server = server
+
+
+ def checkSyncToState(self, state, interval):
+ self.debug("Waiting for background thread to update powerstate to " + state)
+ time.sleep(1 + int(interval)*2/1000) # interval is in ms
+ response = self.getHost(hostId=self.getHost().id).outofbandmanagement
+ self.assertEqual(response.powerstate, state)
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_configure_invalid_driver(self):
+ """
+ Tests out-of-band management configuration with invalid driver
+ """
+ cmd = self.getOobmConfigCmd()
+ cmd.driver = 'randomDriverThatDoesNotExist'
+ try:
+ response = self.apiclient.configureOutOfBandManagement(cmd)
+ self.fail("Expected an exception to be thrown, failing")
+ except Exception: pass
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_configure_default_driver(self):
+ """
+ Tests out-of-band management configuration with valid data
+ """
+ cmd = self.getOobmConfigCmd()
+ response = self.apiclient.configureOutOfBandManagement(cmd)
+ self.assertEqual(response.hostid, cmd.hostid)
+ self.assertEqual(response.driver, cmd.driver)
+ self.assertEqual(response.address, cmd.address)
+ self.assertEqual(response.port, str(cmd.port))
+ self.assertEqual(response.username, cmd.username)
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_enable_feature_invalid(self):
+ """
+ Tests out-of-band management host enable feature with
+ invalid options
+ """
+ cmd = self.getOobmEnableCmd()
+ cmd.hostid = -1
+ try:
+ response = self.apiclient.enableOutOfBandManagementForHost(cmd)
+ self.fail("Expected an exception to be thrown, failing")
+ except Exception: pass
+
+ try:
+ cmd = enableOutOfBandManagementForCluster.enableOutOfBandManagementForClusterCmd()
+ response = self.apiclient.enableOutOfBandManagementForCluster(cmd)
+ self.fail("Expected an exception to be thrown, failing")
+ except Exception: pass
+
+ try:
+ cmd = enableOutOfBandManagementForZone.enableOutOfBandManagementForZoneCmd()
+ response = self.apiclient.enableOutOfBandManagementForZone(cmd)
+ self.fail("Expected an exception to be thrown, failing")
+ except Exception: pass
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_disable_feature_invalid(self):
+ """
+ Tests out-of-band management host disable feature with
+ invalid options
+ """
+ cmd = self.getOobmDisableCmd()
+ cmd.hostid = -1
+ try:
+ response = self.apiclient.disableOutOfBandManagementForHost(cmd)
+ self.fail("Expected an exception to be thrown, failing")
+ except Exception: pass
+
+ try:
+ cmd = disableOutOfBandManagementForCluster.disableOutOfBandManagementForClusterCmd()
+ response = self.apiclient.disableOutOfBandManagementForCluster(cmd)
+ self.fail("Expected an exception to be thrown, failing")
+ except Exception: pass
+
+ try:
+ cmd = disableOutOfBandManagementForZone.disableOutOfBandManagementForZoneCmd()
+ response = self.apiclient.disableOutOfBandManagementForZone(cmd)
+ self.fail("Expected an exception to be thrown, failing")
+ except Exception: pass
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_enable_feature_valid(self):
+ """
+ Tests out-of-band management host enable feature with
+ valid options
+ """
+ self.apiclient.configureOutOfBandManagement(self.getOobmConfigCmd())
+ cmd = self.getOobmEnableCmd()
+ response = self.apiclient.enableOutOfBandManagementForHost(cmd)
+ self.assertEqual(response.hostid, cmd.hostid)
+ self.assertEqual(response.enabled, True)
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_disable_feature_valid(self):
+ """
+ Tests out-of-band management host disable feature with
+ valid options
+ """
+
+ self.apiclient.configureOutOfBandManagement(self.getOobmConfigCmd())
+ cmd = self.getOobmDisableCmd()
+ response = self.apiclient.disableOutOfBandManagementForHost(cmd)
+ self.assertEqual(response.hostid, cmd.hostid)
+ self.assertEqual(response.enabled, False)
+
+ response = self.getHost(hostId=cmd.hostid).outofbandmanagement
+ self.assertEqual(response.powerstate, 'Disabled')
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_enabledisable_across_clusterzones(self):
+ """
+ Tests out-of-band management enable/disable feature at cluster
+ and zone level sequentially Zone > Cluster > Host
+ """
+ self.configureAndEnableOobm()
+ self.startIpmiServer()
+ bmc = IpmiServerContext().bmc
+ bmc.powerstate = 'off'
+
+ host = self.getHost()
+
+ # Disable at zone level
+ cmd = disableOutOfBandManagementForZone.disableOutOfBandManagementForZoneCmd()
+ cmd.zoneid = host.zoneid
+ response = self.apiclient.disableOutOfBandManagementForZone(cmd)
+
+ # Disable at cluster level
+ cmd = disableOutOfBandManagementForCluster.disableOutOfBandManagementForClusterCmd()
+ cmd.clusterid = host.clusterid
+ response = self.apiclient.disableOutOfBandManagementForCluster(cmd)
+
+ # Disable at host level
+ cmd = disableOutOfBandManagementForHost.disableOutOfBandManagementForHostCmd()
+ cmd.hostid = host.id
+ response = self.apiclient.disableOutOfBandManagementForHost(cmd)
+
+ try:
+ self.issuePowerActionCmd('STATUS')
+ self.fail("Exception was expected, oobm is disabled at zone level")
+ except Exception: pass
+
+ # Enable at zone level
+ cmd = enableOutOfBandManagementForZone.enableOutOfBandManagementForZoneCmd()
+ cmd.zoneid = host.zoneid
+ response = self.apiclient.enableOutOfBandManagementForZone(cmd)
+
+ try:
+ self.issuePowerActionCmd('STATUS')
+ self.fail("Exception was expected, oobm is disabled at cluster level")
+ except Exception: pass
+
+ # Check background thread syncs state to Disabled
+ response = self.getHost(hostId=host.id).outofbandmanagement
+ self.assertEqual(response.powerstate, 'Disabled')
+ self.dbclient.execute("update oobm set power_state='On' where port=%d" % self.getIpmiServerPort())
+ interval = list_configurations(
+ self.apiclient,
+ name='outofbandmanagement.sync.interval'
+ )[0].value
+ self.checkSyncToState('Disabled', interval)
+
+ # Enable at cluster level
+ cmd = enableOutOfBandManagementForCluster.enableOutOfBandManagementForClusterCmd()
+ cmd.clusterid = host.clusterid
+ response = self.apiclient.enableOutOfBandManagementForCluster(cmd)
+
+ try:
+ self.issuePowerActionCmd('STATUS')
+ self.fail("Exception was expected, oobm is disabled at host level")
+ except Exception: pass
+
+ # Enable at host level
+ cmd = enableOutOfBandManagementForHost.enableOutOfBandManagementForHostCmd()
+ cmd.hostid = host.id
+ response = self.apiclient.enableOutOfBandManagementForHost(cmd)
+
+ response = self.issuePowerActionCmd('STATUS')
+ self.assertEqual(response.powerstate, 'Off')
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_issue_power_status(self):
+ """
+ Tests out-of-band management issue power action
+ """
+ self.configureAndEnableOobm()
+ self.startIpmiServer()
+ bmc = IpmiServerContext().bmc
+ bmc.powerstate = 'on'
+ response = self.issuePowerActionCmd('STATUS')
+ self.assertEqual(response.powerstate, 'On')
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_issue_power_on(self):
+ """
+ Tests out-of-band management issue power on action
+ """
+ self.configureAndEnableOobm()
+ self.startIpmiServer()
+ self.issuePowerActionCmd('ON')
+ response = self.issuePowerActionCmd('STATUS')
+ self.assertEqual(response.powerstate, 'On')
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_issue_power_off(self):
+ """
+ Tests out-of-band management issue power off action
+ """
+ self.configureAndEnableOobm()
+ self.startIpmiServer()
+ self.issuePowerActionCmd('OFF')
+ response = self.issuePowerActionCmd('STATUS')
+ self.assertEqual(response.powerstate, 'Off')
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_issue_power_cycle(self):
+ """
+ Tests out-of-band management issue power cycle action
+ """
+ self.configureAndEnableOobm()
+ self.startIpmiServer()
+ self.issuePowerActionCmd('CYCLE')
+ response = self.issuePowerActionCmd('STATUS')
+ self.assertEqual(response.powerstate, 'On')
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_issue_power_reset(self):
+ """
+ Tests out-of-band management issue power reset action
+ """
+ self.configureAndEnableOobm()
+ self.startIpmiServer()
+ self.issuePowerActionCmd('RESET')
+ response = self.issuePowerActionCmd('STATUS')
+ self.assertEqual(response.powerstate, 'On')
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_issue_power_soft(self):
+ """
+ Tests out-of-band management issue power soft action
+ """
+ self.configureAndEnableOobm()
+ self.startIpmiServer()
+ self.issuePowerActionCmd('SOFT')
+ response = self.issuePowerActionCmd('STATUS')
+ self.assertEqual(response.powerstate, 'Off')
--- End diff --
Refactored and fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61806348
--- Diff: api/src/org/apache/cloudstack/outofbandmanagement/OutOfBandManagementService.java ---
@@ -0,0 +1,51 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.outofbandmanagement;
+
+import com.cloud.dc.DataCenter;
+import com.cloud.host.Host;
+import com.cloud.org.Cluster;
+import com.google.common.collect.ImmutableMap;
+import org.apache.cloudstack.api.response.OutOfBandManagementResponse;
+import org.apache.cloudstack.framework.config.ConfigKey;
+
+import java.util.List;
+
+public interface OutOfBandManagementService {
+
+ ConfigKey<Long> OutOfBandManagementActionTimeout = new ConfigKey<Long>("Advanced", Long.class, "outofbandmanagement.action.timeout", "60",
+ "The out of band management action timeout in seconds, configurable by cluster", true, ConfigKey.Scope.Cluster);
+
+ ConfigKey<Long> OutOfBandManagementSyncThreadInterval = new ConfigKey<Long>("Advanced", Long.class, "outofbandmanagement.sync.interval", "300000",
+ "The interval (in milliseconds) when the out-of-band management background sync are retrieved", true, ConfigKey.Scope.Global);
+
+ ConfigKey<Integer> OutOfBandManagementSyncThreadPoolSize = new ConfigKey<Integer>("Advanced", Integer.class, "outofbandmanagement.sync.poolsize", "50",
+ "The out of band management background sync thread pool size", true, ConfigKey.Scope.Global);
+
+ long getId();
+ boolean isOutOfBandManagementEnabled(Host host);
+ void submitBackgroundPowerSyncTask(Host host);
+ boolean transitionPowerStateToDisabled(List<? extends Host> hosts);
--- End diff --
Why is ``hosts`` declared as ``List<? extends Host>`` rather than ``List<Host>``?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-218666241
@jburwell fixed the ProcessRunner issues, please do a final review and LGTM or suggest changes. Thanks.
@nvazquez @swill I've fixed two CI issues (Travis and Jenkins issues) in this PR as well
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61810461
--- Diff: server/src/org/apache/cloudstack/outofbandmanagement/OutOfBandManagementServiceImpl.java ---
@@ -0,0 +1,532 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.outofbandmanagement;
+
+import com.cloud.alert.AlertManager;
+import com.cloud.dc.ClusterDetailsDao;
+import com.cloud.dc.ClusterDetailsVO;
+import com.cloud.dc.DataCenter;
+import com.cloud.dc.DataCenterDetailVO;
+import com.cloud.dc.dao.DataCenterDetailsDao;
+import com.cloud.domain.Domain;
+import com.cloud.event.ActionEvent;
+import com.cloud.event.ActionEventUtils;
+import com.cloud.event.EventTypes;
+import com.cloud.host.Host;
+import com.cloud.host.dao.HostDao;
+import com.cloud.org.Cluster;
+import com.cloud.utils.component.Manager;
+import com.cloud.utils.component.ManagerBase;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.db.Transaction;
+import com.cloud.utils.db.TransactionCallback;
+import com.cloud.utils.db.TransactionStatus;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.fsm.NoTransitionException;
+import com.google.common.base.Strings;
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import com.google.common.collect.ImmutableMap;
+import org.apache.cloudstack.api.response.OutOfBandManagementResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.framework.config.ConfigKey;
+import org.apache.cloudstack.framework.config.Configurable;
+import org.apache.cloudstack.outofbandmanagement.dao.OutOfBandManagementDao;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverChangePasswordCommand;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverPowerCommand;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverResponse;
+import org.apache.cloudstack.utils.identity.ManagementServerNode;
+import org.apache.log4j.Logger;
+import org.springframework.stereotype.Component;
+
+import javax.ejb.Local;
+import javax.inject.Inject;
+import javax.naming.ConfigurationException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+@Component
+@Local(value = {OutOfBandManagementService.class})
+public class OutOfBandManagementServiceImpl extends ManagerBase implements OutOfBandManagementService, Manager, Configurable {
+ public static final Logger LOG = Logger.getLogger(OutOfBandManagementServiceImpl.class);
+
+ @Inject
+ private ClusterDetailsDao clusterDetailsDao;
+ @Inject
+ private DataCenterDetailsDao dataCenterDetailsDao;
+ @Inject
+ private OutOfBandManagementDao outOfBandManagementDao;
+ @Inject
+ private HostDao hostDao;
+ @Inject
+ private AlertManager alertMgr;
+
+ private String name;
+ private long serviceId;
+
+ private List<OutOfBandManagementDriver> outOfBandManagementDrivers = new ArrayList<>();
+ private Map<String, OutOfBandManagementDriver> outOfBandManagementDriversMap = new HashMap<String, OutOfBandManagementDriver>();
+
+ private static final String OOBM_ENABLED_DETAIL = "outOfBandManagementEnabled";
+ private static final int ACQUIRE_GLOBAL_LOCK_TIMEOUT_FOR_HOST = 120;
+
+ private Cache<Long, Long> hostAlertCache;
+ private static ExecutorService backgroundSyncExecutor;
+
+ private String getOutOfBandManagementHostLock(long id) {
+ return "oobm.host." + id;
+ }
+
+ private void initializeDriversMap() {
+ if (outOfBandManagementDriversMap.isEmpty() && outOfBandManagementDrivers != null && outOfBandManagementDrivers.size() > 0) {
+ for (final OutOfBandManagementDriver driver : outOfBandManagementDrivers) {
+ outOfBandManagementDriversMap.put(driver.getName().toLowerCase(), driver);
+ }
+ LOG.debug("Discovered out-of-band management drivers configured in the OutOfBandManagementService");
+ }
+ }
+
+ private OutOfBandManagementDriver getDriver(final OutOfBandManagement outOfBandManagementConfig) {
+ if (!Strings.isNullOrEmpty(outOfBandManagementConfig.getDriver())) {
+ final OutOfBandManagementDriver driver = outOfBandManagementDriversMap.get(outOfBandManagementConfig.getDriver());
+ if (driver != null) {
+ return driver;
+ }
+ }
+ throw new CloudRuntimeException("Configured out-of-band management driver is not available. Aborting any out-of-band management action.");
+ }
+
+ protected OutOfBandManagement updateConfig(final OutOfBandManagement outOfBandManagementConfig, final ImmutableMap<OutOfBandManagement.Option, String> options) {
+ if (outOfBandManagementConfig == null) {
+ throw new CloudRuntimeException("Out-of-band management is not configured for the host. Aborting.");
+ }
+ if (options == null) {
+ return outOfBandManagementConfig;
+ }
+ for (OutOfBandManagement.Option option: options.keySet()) {
+ final String value = options.get(option);
+ if (Strings.isNullOrEmpty(value)) {
+ continue;
+ }
+ switch (option) {
+ case DRIVER:
+ outOfBandManagementConfig.setDriver(value);
+ break;
+ case ADDRESS:
+ outOfBandManagementConfig.setAddress(value);
+ break;
+ case PORT:
+ outOfBandManagementConfig.setPort(Integer.parseInt(value));
+ break;
+ case USERNAME:
+ outOfBandManagementConfig.setUsername(value);
+ break;
+ case PASSWORD:
+ outOfBandManagementConfig.setPassword(value);
+ break;
+ }
+ }
+ return outOfBandManagementConfig;
+ }
+
+ protected ImmutableMap<OutOfBandManagement.Option, String> getOptions(final OutOfBandManagement outOfBandManagementConfig) {
+ final ImmutableMap.Builder<OutOfBandManagement.Option, String> optionsBuilder = ImmutableMap.builder();
+ if (outOfBandManagementConfig == null) {
+ throw new CloudRuntimeException("Out-of-band management is not configured for the host. Aborting.");
+ }
+ for (OutOfBandManagement.Option option: OutOfBandManagement.Option.values()) {
+ String value = null;
+ switch (option) {
+ case DRIVER:
+ value = outOfBandManagementConfig.getDriver();
+ break;
+ case ADDRESS:
+ value = outOfBandManagementConfig.getAddress();
+ break;
+ case PORT:
+ if (outOfBandManagementConfig.getPort() != null) {
+ value = String.valueOf(outOfBandManagementConfig.getPort());
+ }
+ break;
+ case USERNAME:
+ value = outOfBandManagementConfig.getUsername();
+ break;
+ case PASSWORD:
+ value = outOfBandManagementConfig.getPassword();
+ break;
+ }
+ if (value != null) {
+ optionsBuilder.put(option, value);
+ }
+ }
+ return optionsBuilder.build();
+ }
+
+ private void sendAuthError(final Host host, final String message) {
+ try {
+ hostAlertCache.asMap().putIfAbsent(host.getId(), 0L);
+ Long sentCount = hostAlertCache.asMap().get(host.getId());
+ if (sentCount != null && sentCount <= 0) {
+ boolean concurrentUpdateResult = hostAlertCache.asMap().replace(host.getId(), sentCount, sentCount+1L);
+ if (concurrentUpdateResult) {
+ final String subject = String.format("Out-of-band management auth-error detected for host:%d in cluster:%d, zone:%d", host.getId(), host.getClusterId(), host.getDataCenterId());
+ LOG.error(subject + ": " + message);
+ alertMgr.sendAlert(AlertManager.AlertType.ALERT_TYPE_OOBM_AUTH_ERROR, host.getDataCenterId(), host.getPodId(), subject, message);
+ }
+ }
+ } catch (Exception ignored) {
+ }
+ }
+
+ private boolean transitionPowerState(OutOfBandManagement.PowerState.Event event, OutOfBandManagement outOfBandManagementHost) {
+ if (outOfBandManagementHost == null) {
+ return false;
+ }
+ OutOfBandManagement.PowerState currentPowerState = outOfBandManagementHost.getPowerState();
+ try {
+ OutOfBandManagement.PowerState newPowerState = OutOfBandManagement.PowerState.getStateMachine().getNextState(currentPowerState, event);
+ boolean result = outOfBandManagementDao.updateState(currentPowerState, event, newPowerState, outOfBandManagementHost, null);
+ if (result) {
+ final String message = String.format("Transitioned out-of-band management power state from:%s to:%s due to event:%s for the host id:%d", currentPowerState, newPowerState, event, outOfBandManagementHost.getHostId());
+ LOG.debug(message);
+ ActionEventUtils.onActionEvent(CallContext.current().getCallingUserId(), CallContext.current().getCallingAccountId(), Domain.ROOT_DOMAIN,
+ EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_POWERSTATE_TRANSITION, message);
+ }
+ return result;
+ } catch (NoTransitionException ignored) {
+ LOG.trace(String.format("Unable to transition out-of-band management power state for host id=%s for the event=%s and current power state=%s", outOfBandManagementHost.getHostId(), event, currentPowerState));
+ }
+ return false;
+ }
+
+ private boolean isOutOfBandManagementEnabledForZone(Long zoneId) {
+ if (zoneId == null) {
+ return true;
+ }
+ final DataCenterDetailVO zoneDetails = dataCenterDetailsDao.findDetail(zoneId, OOBM_ENABLED_DETAIL);
+ if (zoneDetails != null && !Strings.isNullOrEmpty(zoneDetails.getValue()) && !Boolean.valueOf(zoneDetails.getValue())) {
+ return false;
+ }
+ return true;
+ }
+
+ private boolean isOutOfBandManagementEnabledForCluster(Long clusterId) {
+ if (clusterId == null) {
+ return true;
+ }
+ final ClusterDetailsVO clusterDetails = clusterDetailsDao.findDetail(clusterId, OOBM_ENABLED_DETAIL);
+ if (clusterDetails != null && !Strings.isNullOrEmpty(clusterDetails.getValue()) && !Boolean.valueOf(clusterDetails.getValue())) {
+ return false;
+ }
+ return true;
+ }
+
+ private boolean isOutOfBandManagementEnabledForHost(Long hostId) {
+ if (hostId == null) {
+ return false;
+ }
+ final OutOfBandManagement outOfBandManagementConfig = outOfBandManagementDao.findByHost(hostId);
+ if (outOfBandManagementConfig == null || !outOfBandManagementConfig.isEnabled()) {
+ return false;
+ }
+ return true;
+ }
+
+ private void checkOutOfBandManagementEnabledByZoneClusterHost(final Host host) {
+ if (!isOutOfBandManagementEnabledForZone(host.getDataCenterId())) {
+ throw new CloudRuntimeException("Out-of-band management is disabled for the host's zone. Aborting Operation.");
+ }
+ if (!isOutOfBandManagementEnabledForCluster(host.getClusterId())) {
+ throw new CloudRuntimeException("Out-of-band management is disabled for the host's cluster. Aborting Operation.");
+ }
+ if (!isOutOfBandManagementEnabledForHost(host.getId())) {
+ throw new CloudRuntimeException("Out-of-band management is disabled or not configured for the host. Aborting Operation.");
+ }
+ }
+
+ public boolean isOutOfBandManagementEnabled(final Host host) {
+ return isOutOfBandManagementEnabledForZone(host.getDataCenterId())
+ && isOutOfBandManagementEnabledForCluster(host.getClusterId())
+ && isOutOfBandManagementEnabledForHost(host.getId());
+ }
+
+ public boolean transitionPowerStateToDisabled(List<? extends Host> hosts) {
+ boolean result = true;
+ for (Host host : hosts) {
+ result = result && transitionPowerState(OutOfBandManagement.PowerState.Event.Disabled,
+ outOfBandManagementDao.findByHost(host.getId()));
+ }
+ return result;
+ }
+
+ public void submitBackgroundPowerSyncTask(final Host host) {
+ if (host != null) {
+ // Note: This is a blocking queue based executor
+ backgroundSyncExecutor.submit(new OutOfBandManagementBackgroundTask(this, host, OutOfBandManagement.PowerOperation.STATUS));
+ }
+ }
+
+ private OutOfBandManagementResponse buildEnableDisableResponse(final boolean enabled) {
+ final OutOfBandManagementResponse response = new OutOfBandManagementResponse();
+ response.setEnabled(enabled);
+ response.setSuccess(true);
+ return response;
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_ENABLEDISABLE, eventDescription = "enabling/disabling out-of-band management on a zone")
+ public OutOfBandManagementResponse enableDisableOutOfBandManagement(final DataCenter zone, final boolean enabled) {
+ dataCenterDetailsDao.persist(zone.getId(), OOBM_ENABLED_DETAIL, String.valueOf(enabled));
+ if (!enabled) {
+ transitionPowerStateToDisabled(hostDao.findByDataCenterId(zone.getId()));
+ }
+ return buildEnableDisableResponse(enabled);
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_ENABLEDISABLE, eventDescription = "enabling/disabling out-of-band management on a cluster")
+ public OutOfBandManagementResponse enableDisableOutOfBandManagement(final Cluster cluster, final boolean enabled) {
+ clusterDetailsDao.persist(cluster.getId(), OOBM_ENABLED_DETAIL, String.valueOf(enabled));
+ if (!enabled) {
+ transitionPowerStateToDisabled(hostDao.findByClusterId(cluster.getId()));
+ }
+ return buildEnableDisableResponse(enabled);
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_ENABLEDISABLE, eventDescription = "enabling/disabling out-of-band management on a host")
+ public OutOfBandManagementResponse enableDisableOutOfBandManagement(final Host host, final boolean enabled) {
+ final OutOfBandManagement outOfBandManagementConfig = outOfBandManagementDao.findByHost(host.getId());
+ if (outOfBandManagementConfig == null) {
+ final OutOfBandManagementResponse response = new OutOfBandManagementResponse(null);
+ response.setSuccess(false);
+ response.setResultDescription("Out-of-band management is not configured for the host. Please configure the host before enabling/disabling it.");
+ return response;
+ }
+ hostAlertCache.invalidate(host.getId());
+ outOfBandManagementConfig.setEnabled(enabled);
+ boolean updateResult = outOfBandManagementDao.update(outOfBandManagementConfig.getId(), (OutOfBandManagementVO) outOfBandManagementConfig);
+ if (updateResult && !enabled) {
+ transitionPowerStateToDisabled(Collections.singletonList(host));
+ }
+ return buildEnableDisableResponse(enabled && updateResult);
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_CONFIGURE, eventDescription = "updating out-of-band management configuration")
+ public OutOfBandManagementResponse configureOutOfBandManagement(final Host host, final ImmutableMap<OutOfBandManagement.Option, String> options) {
+ OutOfBandManagement outOfBandManagementConfig = outOfBandManagementDao.findByHost(host.getId());
+ if (outOfBandManagementConfig == null) {
+ outOfBandManagementConfig = outOfBandManagementDao.persist(new OutOfBandManagementVO(host.getId()));
+ }
+ outOfBandManagementConfig = updateConfig(outOfBandManagementConfig, options);
+ if (Strings.isNullOrEmpty(outOfBandManagementConfig.getDriver()) || !outOfBandManagementDriversMap.containsKey(outOfBandManagementConfig.getDriver().toLowerCase())) {
+ throw new CloudRuntimeException("Out-of-band management driver is not available. Please provide a valid driver name.");
+ }
+
+ boolean updatedConfig = outOfBandManagementDao.update(outOfBandManagementConfig.getId(), (OutOfBandManagementVO) outOfBandManagementConfig);
+ CallContext.current().setEventDetails("host id:" + host.getId() + " configuration:" + outOfBandManagementConfig.getAddress() + ":" + outOfBandManagementConfig.getPort());
+
+ if (!updatedConfig) {
+ throw new CloudRuntimeException("Failed to update out-of-band management config for the host in the database.");
+ }
+
+ String result = "Out-of-band management successfully configured for the host";
+ LOG.debug(result);
+
+ final OutOfBandManagementResponse response = new OutOfBandManagementResponse(outOfBandManagementDao.findByHost(host.getId()));
+ response.setResultDescription(result);
+ response.setSuccess(true);
+ return response;
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_ACTION, eventDescription = "issuing Host out-of-band management action", async = true)
+ public OutOfBandManagementResponse executeOutOfBandManagementPowerOperation(final Host host, final OutOfBandManagement.PowerOperation powerOperation, final Long timeout) {
+ checkOutOfBandManagementEnabledByZoneClusterHost(host);
+ final OutOfBandManagement outOfBandManagementConfig = outOfBandManagementDao.findByHost(host.getId());
+ final ImmutableMap<OutOfBandManagement.Option, String> options = getOptions(outOfBandManagementConfig);
+ final OutOfBandManagementDriver driver = getDriver(outOfBandManagementConfig);
+
+ Long actionTimeOut = timeout;
+ if (actionTimeOut == null) {
+ actionTimeOut = OutOfBandManagementActionTimeout.valueIn(host.getClusterId());
+ }
+
+ final OutOfBandManagementDriverPowerCommand cmd = new OutOfBandManagementDriverPowerCommand(options, actionTimeOut, powerOperation);
+ final OutOfBandManagementDriverResponse driverResponse = driver.execute(cmd);
+
+ if (driverResponse == null) {
+ throw new CloudRuntimeException(String.format("Out-of-band Management action (%s) on host (%s) failed due to no response from the driver", powerOperation, host.getUuid()));
+ }
+
+ if (powerOperation.equals(OutOfBandManagement.PowerOperation.STATUS)) {
+ transitionPowerState(driverResponse.toEvent(), outOfBandManagementConfig);
+ }
+
+ if (!driverResponse.isSuccess()) {
+ String errorMessage = String.format("Out-of-band Management action (%s) on host (%s) failed with error: %s", powerOperation, host.getUuid(), driverResponse.getError());
+ if (driverResponse.hasAuthFailure()) {
+ errorMessage = String.format("Out-of-band Management action (%s) on host (%s) failed due to authentication error: %s. Please check configured credentials.", powerOperation, host.getUuid(), driverResponse.getError());
+ sendAuthError(host, errorMessage);
+ }
+ if (!powerOperation.equals(OutOfBandManagement.PowerOperation.STATUS)) {
+ LOG.debug(errorMessage);
+ }
+ throw new CloudRuntimeException(errorMessage);
+ }
+
+ final OutOfBandManagementResponse response = new OutOfBandManagementResponse(outOfBandManagementDao.findByHost(host.getId()));
+ response.setSuccess(driverResponse.isSuccess());
+ response.setResultDescription(driverResponse.getResult());
+ response.setId(host.getUuid());
+ response.setOutOfBandManagementAction(powerOperation.toString());
+ return response;
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_CHANGE_PASSWORD, eventDescription = "updating out-of-band management password")
+ public OutOfBandManagementResponse changeOutOfBandManagementPassword(final Host host, final String newPassword) {
+ checkOutOfBandManagementEnabledByZoneClusterHost(host);
+ if (Strings.isNullOrEmpty(newPassword)) {
+ throw new CloudRuntimeException(String.format("Cannot change out-of-band management password as provided new-password is null or empty for the host %s.", host.getUuid()));
+ }
+ GlobalLock outOfBandManagementHostLock = GlobalLock.getInternLock(getOutOfBandManagementHostLock(host.getId()));
+ try {
+ if (outOfBandManagementHostLock.lock(ACQUIRE_GLOBAL_LOCK_TIMEOUT_FOR_HOST)) {
+ try {
+ final OutOfBandManagement outOfBandManagementConfig = outOfBandManagementDao.findByHost(host.getId());
+
+ final ImmutableMap<OutOfBandManagement.Option, String> options = getOptions(outOfBandManagementConfig);
+ if (!(options.containsKey(OutOfBandManagement.Option.PASSWORD) && !Strings.isNullOrEmpty(options.get(OutOfBandManagement.Option.PASSWORD)))) {
+ throw new CloudRuntimeException(String.format("Cannot change out-of-band management password as we've no previously configured password for the host %s.", host.getUuid()));
+ }
+ final OutOfBandManagementDriver driver = getDriver(outOfBandManagementConfig);
+
+ final OutOfBandManagementDriverChangePasswordCommand cmd = new OutOfBandManagementDriverChangePasswordCommand(options, OutOfBandManagementActionTimeout.valueIn(host.getClusterId()), newPassword);
+ final OutOfBandManagementDriverResponse driverResponse;
+ try {
--- End diff --
Why isn't the enclosing ``try`` block sufficient to handle this exception?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61780914
--- Diff: engine/schema/src/org/apache/cloudstack/outofbandmanagement/dao/OutOfBandManagementDaoImpl.java ---
@@ -0,0 +1,163 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.outofbandmanagement.dao;
+
+import com.cloud.utils.DateUtil;
+import com.cloud.utils.db.Attribute;
+import com.cloud.utils.db.DB;
+import com.cloud.utils.db.Filter;
+import com.cloud.utils.db.GenericDaoBase;
+import com.cloud.utils.db.SearchBuilder;
+import com.cloud.utils.db.SearchCriteria;
+import com.cloud.utils.db.TransactionLegacy;
+import com.cloud.utils.db.UpdateBuilder;
+import com.cloud.utils.exception.CloudRuntimeException;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagement;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagementVO;
+import org.apache.log4j.Logger;
+import org.springframework.stereotype.Component;
+
+import javax.ejb.Local;
+import java.sql.PreparedStatement;
+import java.sql.SQLException;
+import java.util.List;
+
+@DB
+@Component
+@Local(value = {OutOfBandManagementDao.class})
+public class OutOfBandManagementDaoImpl extends GenericDaoBase<OutOfBandManagementVO, Long> implements OutOfBandManagementDao {
+ private static final Logger LOG = Logger.getLogger(OutOfBandManagementDaoImpl.class);
+
+ private SearchBuilder<OutOfBandManagementVO> HostSearch;
+ private SearchBuilder<OutOfBandManagementVO> ManagementServerSearch;
+ private SearchBuilder<OutOfBandManagementVO> OutOfBandManagementOwnerSearch;
+ private SearchBuilder<OutOfBandManagementVO> StateUpdateSearch;
+
+ private Attribute PowerStateAttr;
+ private Attribute MsIdAttr;
+ private Attribute UpdateTimeAttr;
+
+ public OutOfBandManagementDaoImpl() {
+ super();
+
+ HostSearch = createSearchBuilder();
+ HostSearch.and("hostId", HostSearch.entity().getHostId(), SearchCriteria.Op.EQ);
+ HostSearch.done();
+
+ ManagementServerSearch = createSearchBuilder();
+ ManagementServerSearch.and("server", ManagementServerSearch.entity().getManagementServerId(), SearchCriteria.Op.EQ);
+ ManagementServerSearch.done();
+
+ OutOfBandManagementOwnerSearch = createSearchBuilder();
+ OutOfBandManagementOwnerSearch.and("server", OutOfBandManagementOwnerSearch.entity().getManagementServerId(), SearchCriteria.Op.EQ);
+ OutOfBandManagementOwnerSearch.or("serverNull", OutOfBandManagementOwnerSearch.entity().getManagementServerId(), SearchCriteria.Op.NULL);
+ OutOfBandManagementOwnerSearch.done();
+
+ StateUpdateSearch = createSearchBuilder();
+ StateUpdateSearch.and("status", StateUpdateSearch.entity().getPowerState(), SearchCriteria.Op.EQ);
+ StateUpdateSearch.and("id", StateUpdateSearch.entity().getId(), SearchCriteria.Op.EQ);
+ StateUpdateSearch.and("update", StateUpdateSearch.entity().getUpdateCount(), SearchCriteria.Op.EQ);
+ StateUpdateSearch.done();
+
+ PowerStateAttr = _allAttributes.get("powerState");
+ MsIdAttr = _allAttributes.get("managementServerId");
+ UpdateTimeAttr = _allAttributes.get("updateTime");
+ assert (PowerStateAttr != null && MsIdAttr != null && UpdateTimeAttr != null) : "Couldn't find one of these attributes";
+ }
+
+ @Override
+ public OutOfBandManagement findByHost(long hostId) {
+ SearchCriteria<OutOfBandManagementVO> sc = HostSearch.create("hostId", hostId);
+ return findOneBy(sc);
+ }
+
+ @Override
+ public List<OutOfBandManagementVO> findAllByManagementServer(long serverId) {
+ SearchCriteria<OutOfBandManagementVO> sc = OutOfBandManagementOwnerSearch.create();
+ sc.setParameters("server", serverId);
+ return listBy(sc, new Filter(OutOfBandManagementVO.class, "updateTime", true, null, null));
+ }
+
+ private void executeExpireOwnershipSql(final String sql, long resource) {
+ TransactionLegacy txn = TransactionLegacy.currentTxn();
+ try {
+ txn.start();
+ PreparedStatement pstmt = txn.prepareAutoCloseStatement(sql);
+ pstmt.setLong(1, resource);
+ pstmt.executeUpdate();
+ txn.commit();
+ } catch (SQLException e) {
+ txn.rollback();
+ throw new CloudRuntimeException("Unable to reset out-of-band management ownership based on resource:" + resource);
+ }
+ }
+
+ @Override
+ public void expireOutOfBandManagementOwnershipByHours(long hours) {
+ final String resetOwnerSql = "UPDATE oobm set mgmt_server_id=NULL where update_time<= (NOW() - INTERVAL ? HOUR)";
+ executeExpireOwnershipSql(resetOwnerSql, hours);
+ }
+
+ @Override
+ public void expireOutOfBandManagementOwnershipByServer(long serverId) {
+ final String resetOwnerSql = "UPDATE oobm set mgmt_server_id=NULL, power_state=NULL where mgmt_server_id=?";
+ executeExpireOwnershipSql(resetOwnerSql, serverId);
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Expired out-of-band management ownership for hosts owned by management server id:" + serverId);
+ }
+ }
+
+ @Override
+ public boolean updateState(OutOfBandManagement.PowerState oldStatus, OutOfBandManagement.PowerState.Event event, OutOfBandManagement.PowerState newStatus, OutOfBandManagement vo, Object data) {
+ // lock target row from beginning to avoid lock-promotion caused deadlock
+ OutOfBandManagementVO oobmHost = lockRow(vo.getId(), true);
+ if (oobmHost == null) {
+ if (LOG.isTraceEnabled()) {
+ LOG.trace("Failed to lock row to update state for out-of-band management host id: " + vo.getHostId());
+ }
+ return false;
+ }
+
+ Long newManagementServerId = event.getServerId();
+ // Avoid updates when old ownership and state are same as new
+ if (oldStatus == newStatus && (oobmHost.getManagementServerId() != null && oobmHost.getManagementServerId().equals(newManagementServerId))) {
+ return false;
+ }
+
+ if (event == OutOfBandManagement.PowerState.Event.Disabled) {
+ newManagementServerId = null;
+ }
+
+ SearchCriteria<OutOfBandManagementVO> sc = StateUpdateSearch.create();
+ sc.setParameters("status", oldStatus);
+ sc.setParameters("id", oobmHost.getId());
+ sc.setParameters("update", oobmHost.getUpdateCount());
+
+ oobmHost.incrUpdateCount();
+ UpdateBuilder ub = getUpdateBuilder(oobmHost);
+ ub.set(oobmHost, PowerStateAttr, newStatus);
+ ub.set(oobmHost, UpdateTimeAttr, DateUtil.currentGMTTime());
+ ub.set(oobmHost, MsIdAttr, newManagementServerId);
+
+ int result = update(ub, sc, null);
+ if (LOG.isDebugEnabled() && result <= 0) {
+ LOG.debug(String.format("Failed to update out-of-band management power state from:%s to:%s due to event:%s for the host id:%d", oldStatus, newStatus, event, oobmHost.getHostId()));
--- End diff --
Should this message be a ``WARN``?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by swill <gi...@git.apache.org>.
Github user swill commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-216884501
BTW, travis is showing exceptions in OOBM now, so you probably want to review those...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61843327
--- Diff: api/src/org/apache/cloudstack/outofbandmanagement/OutOfBandManagementService.java ---
@@ -0,0 +1,51 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.outofbandmanagement;
+
+import com.cloud.dc.DataCenter;
+import com.cloud.host.Host;
+import com.cloud.org.Cluster;
+import com.google.common.collect.ImmutableMap;
+import org.apache.cloudstack.api.response.OutOfBandManagementResponse;
+import org.apache.cloudstack.framework.config.ConfigKey;
+
+import java.util.List;
+
+public interface OutOfBandManagementService {
+
+ ConfigKey<Long> OutOfBandManagementActionTimeout = new ConfigKey<Long>("Advanced", Long.class, "outofbandmanagement.action.timeout", "60",
+ "The out of band management action timeout in seconds, configurable by cluster", true, ConfigKey.Scope.Cluster);
+
+ ConfigKey<Long> OutOfBandManagementSyncThreadInterval = new ConfigKey<Long>("Advanced", Long.class, "outofbandmanagement.sync.interval", "300000",
+ "The interval (in milliseconds) when the out-of-band management background sync are retrieved", true, ConfigKey.Scope.Global);
+
+ ConfigKey<Integer> OutOfBandManagementSyncThreadPoolSize = new ConfigKey<Integer>("Advanced", Integer.class, "outofbandmanagement.sync.poolsize", "50",
+ "The out of band management background sync thread pool size", true, ConfigKey.Scope.Global);
+
+ long getId();
+ boolean isOutOfBandManagementEnabled(Host host);
+ void submitBackgroundPowerSyncTask(Host host);
+ boolean transitionPowerStateToDisabled(List<? extends Host> hosts);
+
+ OutOfBandManagementResponse enableDisableOutOfBandManagement(DataCenter zone, boolean enabled);
+ OutOfBandManagementResponse enableDisableOutOfBandManagement(Cluster cluster, boolean enabled);
+ OutOfBandManagementResponse enableDisableOutOfBandManagement(Host host, boolean enabled);
--- End diff --
They handle both enable/disable operations. It indicates that they handle both, based on the provided enabled boolean.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61780464
--- Diff: engine/schema/src/org/apache/cloudstack/outofbandmanagement/dao/OutOfBandManagementDaoImpl.java ---
@@ -0,0 +1,163 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.outofbandmanagement.dao;
+
+import com.cloud.utils.DateUtil;
+import com.cloud.utils.db.Attribute;
+import com.cloud.utils.db.DB;
+import com.cloud.utils.db.Filter;
+import com.cloud.utils.db.GenericDaoBase;
+import com.cloud.utils.db.SearchBuilder;
+import com.cloud.utils.db.SearchCriteria;
+import com.cloud.utils.db.TransactionLegacy;
+import com.cloud.utils.db.UpdateBuilder;
+import com.cloud.utils.exception.CloudRuntimeException;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagement;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagementVO;
+import org.apache.log4j.Logger;
+import org.springframework.stereotype.Component;
+
+import javax.ejb.Local;
+import java.sql.PreparedStatement;
+import java.sql.SQLException;
+import java.util.List;
+
+@DB
+@Component
+@Local(value = {OutOfBandManagementDao.class})
+public class OutOfBandManagementDaoImpl extends GenericDaoBase<OutOfBandManagementVO, Long> implements OutOfBandManagementDao {
+ private static final Logger LOG = Logger.getLogger(OutOfBandManagementDaoImpl.class);
+
+ private SearchBuilder<OutOfBandManagementVO> HostSearch;
+ private SearchBuilder<OutOfBandManagementVO> ManagementServerSearch;
+ private SearchBuilder<OutOfBandManagementVO> OutOfBandManagementOwnerSearch;
+ private SearchBuilder<OutOfBandManagementVO> StateUpdateSearch;
+
+ private Attribute PowerStateAttr;
+ private Attribute MsIdAttr;
+ private Attribute UpdateTimeAttr;
+
+ public OutOfBandManagementDaoImpl() {
+ super();
+
+ HostSearch = createSearchBuilder();
+ HostSearch.and("hostId", HostSearch.entity().getHostId(), SearchCriteria.Op.EQ);
+ HostSearch.done();
+
+ ManagementServerSearch = createSearchBuilder();
+ ManagementServerSearch.and("server", ManagementServerSearch.entity().getManagementServerId(), SearchCriteria.Op.EQ);
+ ManagementServerSearch.done();
+
+ OutOfBandManagementOwnerSearch = createSearchBuilder();
+ OutOfBandManagementOwnerSearch.and("server", OutOfBandManagementOwnerSearch.entity().getManagementServerId(), SearchCriteria.Op.EQ);
+ OutOfBandManagementOwnerSearch.or("serverNull", OutOfBandManagementOwnerSearch.entity().getManagementServerId(), SearchCriteria.Op.NULL);
+ OutOfBandManagementOwnerSearch.done();
+
+ StateUpdateSearch = createSearchBuilder();
+ StateUpdateSearch.and("status", StateUpdateSearch.entity().getPowerState(), SearchCriteria.Op.EQ);
+ StateUpdateSearch.and("id", StateUpdateSearch.entity().getId(), SearchCriteria.Op.EQ);
+ StateUpdateSearch.and("update", StateUpdateSearch.entity().getUpdateCount(), SearchCriteria.Op.EQ);
+ StateUpdateSearch.done();
+
+ PowerStateAttr = _allAttributes.get("powerState");
+ MsIdAttr = _allAttributes.get("managementServerId");
+ UpdateTimeAttr = _allAttributes.get("updateTime");
+ assert (PowerStateAttr != null && MsIdAttr != null && UpdateTimeAttr != null) : "Couldn't find one of these attributes";
+ }
+
+ @Override
+ public OutOfBandManagement findByHost(long hostId) {
+ SearchCriteria<OutOfBandManagementVO> sc = HostSearch.create("hostId", hostId);
+ return findOneBy(sc);
+ }
+
+ @Override
+ public List<OutOfBandManagementVO> findAllByManagementServer(long serverId) {
+ SearchCriteria<OutOfBandManagementVO> sc = OutOfBandManagementOwnerSearch.create();
+ sc.setParameters("server", serverId);
+ return listBy(sc, new Filter(OutOfBandManagementVO.class, "updateTime", true, null, null));
+ }
+
+ private void executeExpireOwnershipSql(final String sql, long resource) {
+ TransactionLegacy txn = TransactionLegacy.currentTxn();
+ try {
--- End diff --
To ensure that the transaction is properly closed, declare as a try with resources.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r62616608
--- Diff: utils/src/main/java/org/apache/cloudstack/utils/process/ProcessRunner.java ---
@@ -0,0 +1,112 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+
+package org.apache.cloudstack.utils.process;
+
+import com.cloud.utils.concurrency.NamedThreadFactory;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Strings;
+import org.apache.log4j.Logger;
+import org.joda.time.Duration;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.List;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+public class ProcessRunner {
+ public static final Logger LOG = Logger.getLogger(ProcessRunner.class);
+
+ private static final ExecutorService processExecutor = Executors.newCachedThreadPool(new NamedThreadFactory("ProcessRunner"));
+
+ private static String readStream(final InputStream inputStream) {
+ String text = null;
+ try {
+ final BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(inputStream));
+ String line;
--- End diff --
Fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-218677864
tag:mergeready
/cc @swill all green now
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r62570751
--- Diff: utils/src/main/java/org/apache/cloudstack/utils/process/ProcessResult.java ---
@@ -0,0 +1,46 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.utils.process;
+
+public class ProcessResult {
--- End diff --
Could this class be collapsed as a ``static final`` class in ``ProcessRunner``? Also, the class should be ``final`` and the constructor to should either be default (standalone class) or ``private`` (static inner class).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61844518
--- Diff: api/src/org/apache/cloudstack/outofbandmanagement/driver/OutOfBandManagementDriverPowerCommand.java ---
@@ -0,0 +1,33 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.outofbandmanagement.driver;
+
+import com.google.common.collect.ImmutableMap;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagement;
+
+public class OutOfBandManagementDriverPowerCommand extends OutOfBandManagementDriverCommand {
--- End diff --
Fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61846633
--- Diff: server/src/org/apache/cloudstack/outofbandmanagement/OutOfBandManagementServiceImpl.java ---
@@ -0,0 +1,532 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.outofbandmanagement;
+
+import com.cloud.alert.AlertManager;
+import com.cloud.dc.ClusterDetailsDao;
+import com.cloud.dc.ClusterDetailsVO;
+import com.cloud.dc.DataCenter;
+import com.cloud.dc.DataCenterDetailVO;
+import com.cloud.dc.dao.DataCenterDetailsDao;
+import com.cloud.domain.Domain;
+import com.cloud.event.ActionEvent;
+import com.cloud.event.ActionEventUtils;
+import com.cloud.event.EventTypes;
+import com.cloud.host.Host;
+import com.cloud.host.dao.HostDao;
+import com.cloud.org.Cluster;
+import com.cloud.utils.component.Manager;
+import com.cloud.utils.component.ManagerBase;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.db.Transaction;
+import com.cloud.utils.db.TransactionCallback;
+import com.cloud.utils.db.TransactionStatus;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.fsm.NoTransitionException;
+import com.google.common.base.Strings;
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import com.google.common.collect.ImmutableMap;
+import org.apache.cloudstack.api.response.OutOfBandManagementResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.framework.config.ConfigKey;
+import org.apache.cloudstack.framework.config.Configurable;
+import org.apache.cloudstack.outofbandmanagement.dao.OutOfBandManagementDao;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverChangePasswordCommand;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverPowerCommand;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverResponse;
+import org.apache.cloudstack.utils.identity.ManagementServerNode;
+import org.apache.log4j.Logger;
+import org.springframework.stereotype.Component;
+
+import javax.ejb.Local;
+import javax.inject.Inject;
+import javax.naming.ConfigurationException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+@Component
+@Local(value = {OutOfBandManagementService.class})
+public class OutOfBandManagementServiceImpl extends ManagerBase implements OutOfBandManagementService, Manager, Configurable {
+ public static final Logger LOG = Logger.getLogger(OutOfBandManagementServiceImpl.class);
+
+ @Inject
+ private ClusterDetailsDao clusterDetailsDao;
+ @Inject
+ private DataCenterDetailsDao dataCenterDetailsDao;
+ @Inject
+ private OutOfBandManagementDao outOfBandManagementDao;
+ @Inject
+ private HostDao hostDao;
+ @Inject
+ private AlertManager alertMgr;
+
+ private String name;
+ private long serviceId;
+
+ private List<OutOfBandManagementDriver> outOfBandManagementDrivers = new ArrayList<>();
+ private Map<String, OutOfBandManagementDriver> outOfBandManagementDriversMap = new HashMap<String, OutOfBandManagementDriver>();
+
+ private static final String OOBM_ENABLED_DETAIL = "outOfBandManagementEnabled";
+ private static final int ACQUIRE_GLOBAL_LOCK_TIMEOUT_FOR_HOST = 120;
+
+ private Cache<Long, Long> hostAlertCache;
+ private static ExecutorService backgroundSyncExecutor;
+
+ private String getOutOfBandManagementHostLock(long id) {
+ return "oobm.host." + id;
+ }
+
+ private void initializeDriversMap() {
+ if (outOfBandManagementDriversMap.isEmpty() && outOfBandManagementDrivers != null && outOfBandManagementDrivers.size() > 0) {
+ for (final OutOfBandManagementDriver driver : outOfBandManagementDrivers) {
+ outOfBandManagementDriversMap.put(driver.getName().toLowerCase(), driver);
+ }
+ LOG.debug("Discovered out-of-band management drivers configured in the OutOfBandManagementService");
+ }
+ }
+
+ private OutOfBandManagementDriver getDriver(final OutOfBandManagement outOfBandManagementConfig) {
+ if (!Strings.isNullOrEmpty(outOfBandManagementConfig.getDriver())) {
+ final OutOfBandManagementDriver driver = outOfBandManagementDriversMap.get(outOfBandManagementConfig.getDriver());
+ if (driver != null) {
+ return driver;
+ }
+ }
+ throw new CloudRuntimeException("Configured out-of-band management driver is not available. Aborting any out-of-band management action.");
+ }
+
+ protected OutOfBandManagement updateConfig(final OutOfBandManagement outOfBandManagementConfig, final ImmutableMap<OutOfBandManagement.Option, String> options) {
+ if (outOfBandManagementConfig == null) {
+ throw new CloudRuntimeException("Out-of-band management is not configured for the host. Aborting.");
+ }
+ if (options == null) {
+ return outOfBandManagementConfig;
+ }
+ for (OutOfBandManagement.Option option: options.keySet()) {
+ final String value = options.get(option);
+ if (Strings.isNullOrEmpty(value)) {
+ continue;
+ }
+ switch (option) {
+ case DRIVER:
+ outOfBandManagementConfig.setDriver(value);
+ break;
+ case ADDRESS:
+ outOfBandManagementConfig.setAddress(value);
+ break;
+ case PORT:
+ outOfBandManagementConfig.setPort(Integer.parseInt(value));
+ break;
+ case USERNAME:
+ outOfBandManagementConfig.setUsername(value);
+ break;
+ case PASSWORD:
+ outOfBandManagementConfig.setPassword(value);
+ break;
+ }
+ }
+ return outOfBandManagementConfig;
+ }
+
+ protected ImmutableMap<OutOfBandManagement.Option, String> getOptions(final OutOfBandManagement outOfBandManagementConfig) {
+ final ImmutableMap.Builder<OutOfBandManagement.Option, String> optionsBuilder = ImmutableMap.builder();
+ if (outOfBandManagementConfig == null) {
+ throw new CloudRuntimeException("Out-of-band management is not configured for the host. Aborting.");
+ }
+ for (OutOfBandManagement.Option option: OutOfBandManagement.Option.values()) {
+ String value = null;
+ switch (option) {
+ case DRIVER:
+ value = outOfBandManagementConfig.getDriver();
+ break;
+ case ADDRESS:
+ value = outOfBandManagementConfig.getAddress();
+ break;
+ case PORT:
+ if (outOfBandManagementConfig.getPort() != null) {
+ value = String.valueOf(outOfBandManagementConfig.getPort());
+ }
+ break;
+ case USERNAME:
+ value = outOfBandManagementConfig.getUsername();
+ break;
+ case PASSWORD:
+ value = outOfBandManagementConfig.getPassword();
+ break;
+ }
+ if (value != null) {
+ optionsBuilder.put(option, value);
+ }
+ }
+ return optionsBuilder.build();
+ }
+
+ private void sendAuthError(final Host host, final String message) {
+ try {
+ hostAlertCache.asMap().putIfAbsent(host.getId(), 0L);
+ Long sentCount = hostAlertCache.asMap().get(host.getId());
+ if (sentCount != null && sentCount <= 0) {
+ boolean concurrentUpdateResult = hostAlertCache.asMap().replace(host.getId(), sentCount, sentCount+1L);
+ if (concurrentUpdateResult) {
+ final String subject = String.format("Out-of-band management auth-error detected for host:%d in cluster:%d, zone:%d", host.getId(), host.getClusterId(), host.getDataCenterId());
+ LOG.error(subject + ": " + message);
+ alertMgr.sendAlert(AlertManager.AlertType.ALERT_TYPE_OOBM_AUTH_ERROR, host.getDataCenterId(), host.getPodId(), subject, message);
+ }
+ }
+ } catch (Exception ignored) {
+ }
+ }
+
+ private boolean transitionPowerState(OutOfBandManagement.PowerState.Event event, OutOfBandManagement outOfBandManagementHost) {
+ if (outOfBandManagementHost == null) {
+ return false;
+ }
+ OutOfBandManagement.PowerState currentPowerState = outOfBandManagementHost.getPowerState();
+ try {
+ OutOfBandManagement.PowerState newPowerState = OutOfBandManagement.PowerState.getStateMachine().getNextState(currentPowerState, event);
+ boolean result = outOfBandManagementDao.updateState(currentPowerState, event, newPowerState, outOfBandManagementHost, null);
+ if (result) {
+ final String message = String.format("Transitioned out-of-band management power state from:%s to:%s due to event:%s for the host id:%d", currentPowerState, newPowerState, event, outOfBandManagementHost.getHostId());
+ LOG.debug(message);
+ ActionEventUtils.onActionEvent(CallContext.current().getCallingUserId(), CallContext.current().getCallingAccountId(), Domain.ROOT_DOMAIN,
+ EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_POWERSTATE_TRANSITION, message);
+ }
+ return result;
+ } catch (NoTransitionException ignored) {
+ LOG.trace(String.format("Unable to transition out-of-band management power state for host id=%s for the event=%s and current power state=%s", outOfBandManagementHost.getHostId(), event, currentPowerState));
+ }
+ return false;
+ }
+
+ private boolean isOutOfBandManagementEnabledForZone(Long zoneId) {
+ if (zoneId == null) {
+ return true;
+ }
+ final DataCenterDetailVO zoneDetails = dataCenterDetailsDao.findDetail(zoneId, OOBM_ENABLED_DETAIL);
+ if (zoneDetails != null && !Strings.isNullOrEmpty(zoneDetails.getValue()) && !Boolean.valueOf(zoneDetails.getValue())) {
+ return false;
+ }
+ return true;
+ }
+
+ private boolean isOutOfBandManagementEnabledForCluster(Long clusterId) {
+ if (clusterId == null) {
+ return true;
+ }
+ final ClusterDetailsVO clusterDetails = clusterDetailsDao.findDetail(clusterId, OOBM_ENABLED_DETAIL);
+ if (clusterDetails != null && !Strings.isNullOrEmpty(clusterDetails.getValue()) && !Boolean.valueOf(clusterDetails.getValue())) {
+ return false;
+ }
+ return true;
+ }
+
+ private boolean isOutOfBandManagementEnabledForHost(Long hostId) {
+ if (hostId == null) {
+ return false;
+ }
+ final OutOfBandManagement outOfBandManagementConfig = outOfBandManagementDao.findByHost(hostId);
+ if (outOfBandManagementConfig == null || !outOfBandManagementConfig.isEnabled()) {
+ return false;
+ }
+ return true;
+ }
+
+ private void checkOutOfBandManagementEnabledByZoneClusterHost(final Host host) {
+ if (!isOutOfBandManagementEnabledForZone(host.getDataCenterId())) {
+ throw new CloudRuntimeException("Out-of-band management is disabled for the host's zone. Aborting Operation.");
+ }
+ if (!isOutOfBandManagementEnabledForCluster(host.getClusterId())) {
+ throw new CloudRuntimeException("Out-of-band management is disabled for the host's cluster. Aborting Operation.");
+ }
+ if (!isOutOfBandManagementEnabledForHost(host.getId())) {
+ throw new CloudRuntimeException("Out-of-band management is disabled or not configured for the host. Aborting Operation.");
+ }
+ }
+
+ public boolean isOutOfBandManagementEnabled(final Host host) {
+ return isOutOfBandManagementEnabledForZone(host.getDataCenterId())
+ && isOutOfBandManagementEnabledForCluster(host.getClusterId())
+ && isOutOfBandManagementEnabledForHost(host.getId());
+ }
+
+ public boolean transitionPowerStateToDisabled(List<? extends Host> hosts) {
+ boolean result = true;
+ for (Host host : hosts) {
+ result = result && transitionPowerState(OutOfBandManagement.PowerState.Event.Disabled,
+ outOfBandManagementDao.findByHost(host.getId()));
+ }
+ return result;
+ }
+
+ public void submitBackgroundPowerSyncTask(final Host host) {
+ if (host != null) {
+ // Note: This is a blocking queue based executor
+ backgroundSyncExecutor.submit(new OutOfBandManagementBackgroundTask(this, host, OutOfBandManagement.PowerOperation.STATUS));
+ }
+ }
+
+ private OutOfBandManagementResponse buildEnableDisableResponse(final boolean enabled) {
+ final OutOfBandManagementResponse response = new OutOfBandManagementResponse();
+ response.setEnabled(enabled);
+ response.setSuccess(true);
+ return response;
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_ENABLEDISABLE, eventDescription = "enabling/disabling out-of-band management on a zone")
+ public OutOfBandManagementResponse enableDisableOutOfBandManagement(final DataCenter zone, final boolean enabled) {
+ dataCenterDetailsDao.persist(zone.getId(), OOBM_ENABLED_DETAIL, String.valueOf(enabled));
+ if (!enabled) {
+ transitionPowerStateToDisabled(hostDao.findByDataCenterId(zone.getId()));
+ }
+ return buildEnableDisableResponse(enabled);
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_ENABLEDISABLE, eventDescription = "enabling/disabling out-of-band management on a cluster")
+ public OutOfBandManagementResponse enableDisableOutOfBandManagement(final Cluster cluster, final boolean enabled) {
+ clusterDetailsDao.persist(cluster.getId(), OOBM_ENABLED_DETAIL, String.valueOf(enabled));
+ if (!enabled) {
+ transitionPowerStateToDisabled(hostDao.findByClusterId(cluster.getId()));
+ }
+ return buildEnableDisableResponse(enabled);
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_ENABLEDISABLE, eventDescription = "enabling/disabling out-of-band management on a host")
+ public OutOfBandManagementResponse enableDisableOutOfBandManagement(final Host host, final boolean enabled) {
+ final OutOfBandManagement outOfBandManagementConfig = outOfBandManagementDao.findByHost(host.getId());
+ if (outOfBandManagementConfig == null) {
+ final OutOfBandManagementResponse response = new OutOfBandManagementResponse(null);
+ response.setSuccess(false);
+ response.setResultDescription("Out-of-band management is not configured for the host. Please configure the host before enabling/disabling it.");
+ return response;
+ }
+ hostAlertCache.invalidate(host.getId());
+ outOfBandManagementConfig.setEnabled(enabled);
+ boolean updateResult = outOfBandManagementDao.update(outOfBandManagementConfig.getId(), (OutOfBandManagementVO) outOfBandManagementConfig);
+ if (updateResult && !enabled) {
+ transitionPowerStateToDisabled(Collections.singletonList(host));
+ }
+ return buildEnableDisableResponse(enabled && updateResult);
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_CONFIGURE, eventDescription = "updating out-of-band management configuration")
+ public OutOfBandManagementResponse configureOutOfBandManagement(final Host host, final ImmutableMap<OutOfBandManagement.Option, String> options) {
+ OutOfBandManagement outOfBandManagementConfig = outOfBandManagementDao.findByHost(host.getId());
+ if (outOfBandManagementConfig == null) {
+ outOfBandManagementConfig = outOfBandManagementDao.persist(new OutOfBandManagementVO(host.getId()));
+ }
+ outOfBandManagementConfig = updateConfig(outOfBandManagementConfig, options);
+ if (Strings.isNullOrEmpty(outOfBandManagementConfig.getDriver()) || !outOfBandManagementDriversMap.containsKey(outOfBandManagementConfig.getDriver().toLowerCase())) {
+ throw new CloudRuntimeException("Out-of-band management driver is not available. Please provide a valid driver name.");
+ }
+
+ boolean updatedConfig = outOfBandManagementDao.update(outOfBandManagementConfig.getId(), (OutOfBandManagementVO) outOfBandManagementConfig);
+ CallContext.current().setEventDetails("host id:" + host.getId() + " configuration:" + outOfBandManagementConfig.getAddress() + ":" + outOfBandManagementConfig.getPort());
+
+ if (!updatedConfig) {
+ throw new CloudRuntimeException("Failed to update out-of-band management config for the host in the database.");
+ }
+
+ String result = "Out-of-band management successfully configured for the host";
+ LOG.debug(result);
+
+ final OutOfBandManagementResponse response = new OutOfBandManagementResponse(outOfBandManagementDao.findByHost(host.getId()));
+ response.setResultDescription(result);
+ response.setSuccess(true);
+ return response;
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_ACTION, eventDescription = "issuing Host out-of-band management action", async = true)
+ public OutOfBandManagementResponse executeOutOfBandManagementPowerOperation(final Host host, final OutOfBandManagement.PowerOperation powerOperation, final Long timeout) {
+ checkOutOfBandManagementEnabledByZoneClusterHost(host);
--- End diff --
User/admin is trying to execute or perform an operation on a resource that is not configured, therefore an exception is throw.
Only an admin can call this API, there is no way a DoS can be launched. The argument present for this specific API would apply for any non-list API.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61846775
--- Diff: server/src/org/apache/cloudstack/outofbandmanagement/OutOfBandManagementServiceImpl.java ---
@@ -0,0 +1,532 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.outofbandmanagement;
+
+import com.cloud.alert.AlertManager;
+import com.cloud.dc.ClusterDetailsDao;
+import com.cloud.dc.ClusterDetailsVO;
+import com.cloud.dc.DataCenter;
+import com.cloud.dc.DataCenterDetailVO;
+import com.cloud.dc.dao.DataCenterDetailsDao;
+import com.cloud.domain.Domain;
+import com.cloud.event.ActionEvent;
+import com.cloud.event.ActionEventUtils;
+import com.cloud.event.EventTypes;
+import com.cloud.host.Host;
+import com.cloud.host.dao.HostDao;
+import com.cloud.org.Cluster;
+import com.cloud.utils.component.Manager;
+import com.cloud.utils.component.ManagerBase;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.db.Transaction;
+import com.cloud.utils.db.TransactionCallback;
+import com.cloud.utils.db.TransactionStatus;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.fsm.NoTransitionException;
+import com.google.common.base.Strings;
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import com.google.common.collect.ImmutableMap;
+import org.apache.cloudstack.api.response.OutOfBandManagementResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.framework.config.ConfigKey;
+import org.apache.cloudstack.framework.config.Configurable;
+import org.apache.cloudstack.outofbandmanagement.dao.OutOfBandManagementDao;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverChangePasswordCommand;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverPowerCommand;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverResponse;
+import org.apache.cloudstack.utils.identity.ManagementServerNode;
+import org.apache.log4j.Logger;
+import org.springframework.stereotype.Component;
+
+import javax.ejb.Local;
+import javax.inject.Inject;
+import javax.naming.ConfigurationException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+@Component
+@Local(value = {OutOfBandManagementService.class})
+public class OutOfBandManagementServiceImpl extends ManagerBase implements OutOfBandManagementService, Manager, Configurable {
+ public static final Logger LOG = Logger.getLogger(OutOfBandManagementServiceImpl.class);
+
+ @Inject
+ private ClusterDetailsDao clusterDetailsDao;
+ @Inject
+ private DataCenterDetailsDao dataCenterDetailsDao;
+ @Inject
+ private OutOfBandManagementDao outOfBandManagementDao;
+ @Inject
+ private HostDao hostDao;
+ @Inject
+ private AlertManager alertMgr;
+
+ private String name;
+ private long serviceId;
+
+ private List<OutOfBandManagementDriver> outOfBandManagementDrivers = new ArrayList<>();
+ private Map<String, OutOfBandManagementDriver> outOfBandManagementDriversMap = new HashMap<String, OutOfBandManagementDriver>();
+
+ private static final String OOBM_ENABLED_DETAIL = "outOfBandManagementEnabled";
+ private static final int ACQUIRE_GLOBAL_LOCK_TIMEOUT_FOR_HOST = 120;
+
+ private Cache<Long, Long> hostAlertCache;
+ private static ExecutorService backgroundSyncExecutor;
--- End diff --
variables are configured in configure(), we should not declare them final without initializing them.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61842996
--- Diff: api/src/org/apache/cloudstack/api/command/admin/outofbandmanagement/ChangeOutOfBandManagementPasswordCmd.java ---
@@ -0,0 +1,116 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.api.command.admin.outofbandmanagement;
+
+import com.cloud.event.EventTypes;
+import com.cloud.exception.ConcurrentOperationException;
+import com.cloud.exception.InsufficientCapacityException;
+import com.cloud.exception.NetworkRuleConflictException;
+import com.cloud.exception.ResourceAllocationException;
+import com.cloud.exception.ResourceUnavailableException;
+import com.cloud.host.Host;
+import com.google.common.base.Strings;
+import org.apache.cloudstack.acl.RoleType;
+import org.apache.cloudstack.api.APICommand;
+import org.apache.cloudstack.api.ApiConstants;
+import org.apache.cloudstack.api.ApiErrorCode;
+import org.apache.cloudstack.api.BaseAsyncCmd;
+import org.apache.cloudstack.api.Parameter;
+import org.apache.cloudstack.api.ServerApiException;
+import org.apache.cloudstack.api.response.HostResponse;
+import org.apache.cloudstack.api.response.OutOfBandManagementResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagementService;
+
+import javax.inject.Inject;
+
+@APICommand(name = "changeOutOfBandManagementPassword", description = "Changes out-of-band management interface password on the host and updates the interface configuration in CloudStack if the operation succeeds, else reverts the old password",
--- End diff --
fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by DaanHoogland <gi...@git.apache.org>.
Github user DaanHoogland commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-214667168
@rhtyd got ya. It will work like this but then still some tests require the simulator which is hardware in a sense. I'm just thinking, not commenting.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61843016
--- Diff: api/src/org/apache/cloudstack/api/command/admin/outofbandmanagement/DisableOutOfBandManagementForZoneCmd.java ---
@@ -0,0 +1,105 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.api.command.admin.outofbandmanagement;
+
+import com.cloud.dc.DataCenter;
+import com.cloud.event.EventTypes;
+import com.cloud.exception.ConcurrentOperationException;
+import com.cloud.exception.InsufficientCapacityException;
+import com.cloud.exception.NetworkRuleConflictException;
+import com.cloud.exception.ResourceAllocationException;
+import com.cloud.exception.ResourceUnavailableException;
+import org.apache.cloudstack.acl.RoleType;
+import org.apache.cloudstack.api.APICommand;
+import org.apache.cloudstack.api.ApiConstants;
+import org.apache.cloudstack.api.ApiErrorCode;
+import org.apache.cloudstack.api.BaseAsyncCmd;
+import org.apache.cloudstack.api.BaseCmd;
+import org.apache.cloudstack.api.Parameter;
+import org.apache.cloudstack.api.ServerApiException;
+import org.apache.cloudstack.api.response.OutOfBandManagementResponse;
+import org.apache.cloudstack.api.response.ZoneResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagementService;
+
+import javax.inject.Inject;
+
+@APICommand(name = "disableOutOfBandManagementForZone", description = "Disables out-of-band management for a zone",
--- End diff --
fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61843002
--- Diff: api/src/org/apache/cloudstack/api/command/admin/outofbandmanagement/ConfigureOutOfBandManagementCmd.java ---
@@ -0,0 +1,127 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.api.command.admin.outofbandmanagement;
+
+import com.cloud.exception.ConcurrentOperationException;
+import com.cloud.exception.InsufficientCapacityException;
+import com.cloud.exception.NetworkRuleConflictException;
+import com.cloud.exception.ResourceAllocationException;
+import com.cloud.exception.ResourceUnavailableException;
+import com.cloud.host.Host;
+import com.google.common.base.Strings;
+import com.google.common.collect.ImmutableMap;
+import org.apache.cloudstack.acl.RoleType;
+import org.apache.cloudstack.api.APICommand;
+import org.apache.cloudstack.api.ApiConstants;
+import org.apache.cloudstack.api.ApiErrorCode;
+import org.apache.cloudstack.api.BaseCmd;
+import org.apache.cloudstack.api.Parameter;
+import org.apache.cloudstack.api.ServerApiException;
+import org.apache.cloudstack.api.response.HostResponse;
+import org.apache.cloudstack.api.response.OutOfBandManagementResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagement;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagementService;
+
+import javax.inject.Inject;
+
+@APICommand(name = "configureOutOfBandManagement", description = "Configures a host's out-of-band management interface",
--- End diff --
fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61846972
--- Diff: api/src/org/apache/cloudstack/outofbandmanagement/OutOfBandManagementService.java ---
@@ -0,0 +1,51 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.outofbandmanagement;
+
+import com.cloud.dc.DataCenter;
+import com.cloud.host.Host;
+import com.cloud.org.Cluster;
+import com.google.common.collect.ImmutableMap;
+import org.apache.cloudstack.api.response.OutOfBandManagementResponse;
+import org.apache.cloudstack.framework.config.ConfigKey;
+
+import java.util.List;
+
+public interface OutOfBandManagementService {
+
+ ConfigKey<Long> OutOfBandManagementActionTimeout = new ConfigKey<Long>("Advanced", Long.class, "outofbandmanagement.action.timeout", "60",
+ "The out of band management action timeout in seconds, configurable by cluster", true, ConfigKey.Scope.Cluster);
+
+ ConfigKey<Long> OutOfBandManagementSyncThreadInterval = new ConfigKey<Long>("Advanced", Long.class, "outofbandmanagement.sync.interval", "300000",
+ "The interval (in milliseconds) when the out-of-band management background sync are retrieved", true, ConfigKey.Scope.Global);
+
+ ConfigKey<Integer> OutOfBandManagementSyncThreadPoolSize = new ConfigKey<Integer>("Advanced", Integer.class, "outofbandmanagement.sync.poolsize", "50",
+ "The out of band management background sync thread pool size", true, ConfigKey.Scope.Global);
+
+ long getId();
+ boolean isOutOfBandManagementEnabled(Host host);
+ void submitBackgroundPowerSyncTask(Host host);
+ boolean transitionPowerStateToDisabled(List<? extends Host> hosts);
--- End diff --
There is nothing wrong with the code, it's a valid declaration and usage of generics that accepts any list of concrete items that implement the Host interface; and a classic usage of `PECS` http://stackoverflow.com/questions/2723397/what-is-pecs-producer-extends-consumer-super
Once RBAC PR is merged, I'll use the ListUtils in there that would return a List<Host> along with fix the validateParams with api annotation validators.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61810525
--- Diff: server/src/org/apache/cloudstack/outofbandmanagement/OutOfBandManagementServiceImpl.java ---
@@ -0,0 +1,532 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.outofbandmanagement;
+
+import com.cloud.alert.AlertManager;
+import com.cloud.dc.ClusterDetailsDao;
+import com.cloud.dc.ClusterDetailsVO;
+import com.cloud.dc.DataCenter;
+import com.cloud.dc.DataCenterDetailVO;
+import com.cloud.dc.dao.DataCenterDetailsDao;
+import com.cloud.domain.Domain;
+import com.cloud.event.ActionEvent;
+import com.cloud.event.ActionEventUtils;
+import com.cloud.event.EventTypes;
+import com.cloud.host.Host;
+import com.cloud.host.dao.HostDao;
+import com.cloud.org.Cluster;
+import com.cloud.utils.component.Manager;
+import com.cloud.utils.component.ManagerBase;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.db.Transaction;
+import com.cloud.utils.db.TransactionCallback;
+import com.cloud.utils.db.TransactionStatus;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.fsm.NoTransitionException;
+import com.google.common.base.Strings;
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import com.google.common.collect.ImmutableMap;
+import org.apache.cloudstack.api.response.OutOfBandManagementResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.framework.config.ConfigKey;
+import org.apache.cloudstack.framework.config.Configurable;
+import org.apache.cloudstack.outofbandmanagement.dao.OutOfBandManagementDao;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverChangePasswordCommand;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverPowerCommand;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverResponse;
+import org.apache.cloudstack.utils.identity.ManagementServerNode;
+import org.apache.log4j.Logger;
+import org.springframework.stereotype.Component;
+
+import javax.ejb.Local;
+import javax.inject.Inject;
+import javax.naming.ConfigurationException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+@Component
+@Local(value = {OutOfBandManagementService.class})
+public class OutOfBandManagementServiceImpl extends ManagerBase implements OutOfBandManagementService, Manager, Configurable {
+ public static final Logger LOG = Logger.getLogger(OutOfBandManagementServiceImpl.class);
+
+ @Inject
+ private ClusterDetailsDao clusterDetailsDao;
+ @Inject
+ private DataCenterDetailsDao dataCenterDetailsDao;
+ @Inject
+ private OutOfBandManagementDao outOfBandManagementDao;
+ @Inject
+ private HostDao hostDao;
+ @Inject
+ private AlertManager alertMgr;
+
+ private String name;
+ private long serviceId;
+
+ private List<OutOfBandManagementDriver> outOfBandManagementDrivers = new ArrayList<>();
+ private Map<String, OutOfBandManagementDriver> outOfBandManagementDriversMap = new HashMap<String, OutOfBandManagementDriver>();
+
+ private static final String OOBM_ENABLED_DETAIL = "outOfBandManagementEnabled";
+ private static final int ACQUIRE_GLOBAL_LOCK_TIMEOUT_FOR_HOST = 120;
+
+ private Cache<Long, Long> hostAlertCache;
+ private static ExecutorService backgroundSyncExecutor;
+
+ private String getOutOfBandManagementHostLock(long id) {
+ return "oobm.host." + id;
+ }
+
+ private void initializeDriversMap() {
+ if (outOfBandManagementDriversMap.isEmpty() && outOfBandManagementDrivers != null && outOfBandManagementDrivers.size() > 0) {
+ for (final OutOfBandManagementDriver driver : outOfBandManagementDrivers) {
+ outOfBandManagementDriversMap.put(driver.getName().toLowerCase(), driver);
+ }
+ LOG.debug("Discovered out-of-band management drivers configured in the OutOfBandManagementService");
+ }
+ }
+
+ private OutOfBandManagementDriver getDriver(final OutOfBandManagement outOfBandManagementConfig) {
+ if (!Strings.isNullOrEmpty(outOfBandManagementConfig.getDriver())) {
+ final OutOfBandManagementDriver driver = outOfBandManagementDriversMap.get(outOfBandManagementConfig.getDriver());
+ if (driver != null) {
+ return driver;
+ }
+ }
+ throw new CloudRuntimeException("Configured out-of-band management driver is not available. Aborting any out-of-band management action.");
+ }
+
+ protected OutOfBandManagement updateConfig(final OutOfBandManagement outOfBandManagementConfig, final ImmutableMap<OutOfBandManagement.Option, String> options) {
+ if (outOfBandManagementConfig == null) {
+ throw new CloudRuntimeException("Out-of-band management is not configured for the host. Aborting.");
+ }
+ if (options == null) {
+ return outOfBandManagementConfig;
+ }
+ for (OutOfBandManagement.Option option: options.keySet()) {
+ final String value = options.get(option);
+ if (Strings.isNullOrEmpty(value)) {
+ continue;
+ }
+ switch (option) {
+ case DRIVER:
+ outOfBandManagementConfig.setDriver(value);
+ break;
+ case ADDRESS:
+ outOfBandManagementConfig.setAddress(value);
+ break;
+ case PORT:
+ outOfBandManagementConfig.setPort(Integer.parseInt(value));
+ break;
+ case USERNAME:
+ outOfBandManagementConfig.setUsername(value);
+ break;
+ case PASSWORD:
+ outOfBandManagementConfig.setPassword(value);
+ break;
+ }
+ }
+ return outOfBandManagementConfig;
+ }
+
+ protected ImmutableMap<OutOfBandManagement.Option, String> getOptions(final OutOfBandManagement outOfBandManagementConfig) {
+ final ImmutableMap.Builder<OutOfBandManagement.Option, String> optionsBuilder = ImmutableMap.builder();
+ if (outOfBandManagementConfig == null) {
+ throw new CloudRuntimeException("Out-of-band management is not configured for the host. Aborting.");
+ }
+ for (OutOfBandManagement.Option option: OutOfBandManagement.Option.values()) {
+ String value = null;
+ switch (option) {
+ case DRIVER:
+ value = outOfBandManagementConfig.getDriver();
+ break;
+ case ADDRESS:
+ value = outOfBandManagementConfig.getAddress();
+ break;
+ case PORT:
+ if (outOfBandManagementConfig.getPort() != null) {
+ value = String.valueOf(outOfBandManagementConfig.getPort());
+ }
+ break;
+ case USERNAME:
+ value = outOfBandManagementConfig.getUsername();
+ break;
+ case PASSWORD:
+ value = outOfBandManagementConfig.getPassword();
+ break;
+ }
+ if (value != null) {
+ optionsBuilder.put(option, value);
+ }
+ }
+ return optionsBuilder.build();
+ }
+
+ private void sendAuthError(final Host host, final String message) {
+ try {
+ hostAlertCache.asMap().putIfAbsent(host.getId(), 0L);
+ Long sentCount = hostAlertCache.asMap().get(host.getId());
+ if (sentCount != null && sentCount <= 0) {
+ boolean concurrentUpdateResult = hostAlertCache.asMap().replace(host.getId(), sentCount, sentCount+1L);
+ if (concurrentUpdateResult) {
+ final String subject = String.format("Out-of-band management auth-error detected for host:%d in cluster:%d, zone:%d", host.getId(), host.getClusterId(), host.getDataCenterId());
+ LOG.error(subject + ": " + message);
+ alertMgr.sendAlert(AlertManager.AlertType.ALERT_TYPE_OOBM_AUTH_ERROR, host.getDataCenterId(), host.getPodId(), subject, message);
+ }
+ }
+ } catch (Exception ignored) {
+ }
+ }
+
+ private boolean transitionPowerState(OutOfBandManagement.PowerState.Event event, OutOfBandManagement outOfBandManagementHost) {
+ if (outOfBandManagementHost == null) {
+ return false;
+ }
+ OutOfBandManagement.PowerState currentPowerState = outOfBandManagementHost.getPowerState();
+ try {
+ OutOfBandManagement.PowerState newPowerState = OutOfBandManagement.PowerState.getStateMachine().getNextState(currentPowerState, event);
+ boolean result = outOfBandManagementDao.updateState(currentPowerState, event, newPowerState, outOfBandManagementHost, null);
+ if (result) {
+ final String message = String.format("Transitioned out-of-band management power state from:%s to:%s due to event:%s for the host id:%d", currentPowerState, newPowerState, event, outOfBandManagementHost.getHostId());
+ LOG.debug(message);
+ ActionEventUtils.onActionEvent(CallContext.current().getCallingUserId(), CallContext.current().getCallingAccountId(), Domain.ROOT_DOMAIN,
+ EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_POWERSTATE_TRANSITION, message);
+ }
+ return result;
+ } catch (NoTransitionException ignored) {
+ LOG.trace(String.format("Unable to transition out-of-band management power state for host id=%s for the event=%s and current power state=%s", outOfBandManagementHost.getHostId(), event, currentPowerState));
+ }
+ return false;
+ }
+
+ private boolean isOutOfBandManagementEnabledForZone(Long zoneId) {
+ if (zoneId == null) {
+ return true;
+ }
+ final DataCenterDetailVO zoneDetails = dataCenterDetailsDao.findDetail(zoneId, OOBM_ENABLED_DETAIL);
+ if (zoneDetails != null && !Strings.isNullOrEmpty(zoneDetails.getValue()) && !Boolean.valueOf(zoneDetails.getValue())) {
+ return false;
+ }
+ return true;
+ }
+
+ private boolean isOutOfBandManagementEnabledForCluster(Long clusterId) {
+ if (clusterId == null) {
+ return true;
+ }
+ final ClusterDetailsVO clusterDetails = clusterDetailsDao.findDetail(clusterId, OOBM_ENABLED_DETAIL);
+ if (clusterDetails != null && !Strings.isNullOrEmpty(clusterDetails.getValue()) && !Boolean.valueOf(clusterDetails.getValue())) {
+ return false;
+ }
+ return true;
+ }
+
+ private boolean isOutOfBandManagementEnabledForHost(Long hostId) {
+ if (hostId == null) {
+ return false;
+ }
+ final OutOfBandManagement outOfBandManagementConfig = outOfBandManagementDao.findByHost(hostId);
+ if (outOfBandManagementConfig == null || !outOfBandManagementConfig.isEnabled()) {
+ return false;
+ }
+ return true;
+ }
+
+ private void checkOutOfBandManagementEnabledByZoneClusterHost(final Host host) {
+ if (!isOutOfBandManagementEnabledForZone(host.getDataCenterId())) {
+ throw new CloudRuntimeException("Out-of-band management is disabled for the host's zone. Aborting Operation.");
+ }
+ if (!isOutOfBandManagementEnabledForCluster(host.getClusterId())) {
+ throw new CloudRuntimeException("Out-of-band management is disabled for the host's cluster. Aborting Operation.");
+ }
+ if (!isOutOfBandManagementEnabledForHost(host.getId())) {
+ throw new CloudRuntimeException("Out-of-band management is disabled or not configured for the host. Aborting Operation.");
+ }
+ }
+
+ public boolean isOutOfBandManagementEnabled(final Host host) {
+ return isOutOfBandManagementEnabledForZone(host.getDataCenterId())
+ && isOutOfBandManagementEnabledForCluster(host.getClusterId())
+ && isOutOfBandManagementEnabledForHost(host.getId());
+ }
+
+ public boolean transitionPowerStateToDisabled(List<? extends Host> hosts) {
+ boolean result = true;
+ for (Host host : hosts) {
+ result = result && transitionPowerState(OutOfBandManagement.PowerState.Event.Disabled,
+ outOfBandManagementDao.findByHost(host.getId()));
+ }
+ return result;
+ }
+
+ public void submitBackgroundPowerSyncTask(final Host host) {
+ if (host != null) {
+ // Note: This is a blocking queue based executor
+ backgroundSyncExecutor.submit(new OutOfBandManagementBackgroundTask(this, host, OutOfBandManagement.PowerOperation.STATUS));
+ }
+ }
+
+ private OutOfBandManagementResponse buildEnableDisableResponse(final boolean enabled) {
+ final OutOfBandManagementResponse response = new OutOfBandManagementResponse();
+ response.setEnabled(enabled);
+ response.setSuccess(true);
+ return response;
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_ENABLEDISABLE, eventDescription = "enabling/disabling out-of-band management on a zone")
+ public OutOfBandManagementResponse enableDisableOutOfBandManagement(final DataCenter zone, final boolean enabled) {
+ dataCenterDetailsDao.persist(zone.getId(), OOBM_ENABLED_DETAIL, String.valueOf(enabled));
+ if (!enabled) {
+ transitionPowerStateToDisabled(hostDao.findByDataCenterId(zone.getId()));
+ }
+ return buildEnableDisableResponse(enabled);
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_ENABLEDISABLE, eventDescription = "enabling/disabling out-of-band management on a cluster")
+ public OutOfBandManagementResponse enableDisableOutOfBandManagement(final Cluster cluster, final boolean enabled) {
+ clusterDetailsDao.persist(cluster.getId(), OOBM_ENABLED_DETAIL, String.valueOf(enabled));
+ if (!enabled) {
+ transitionPowerStateToDisabled(hostDao.findByClusterId(cluster.getId()));
+ }
+ return buildEnableDisableResponse(enabled);
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_ENABLEDISABLE, eventDescription = "enabling/disabling out-of-band management on a host")
+ public OutOfBandManagementResponse enableDisableOutOfBandManagement(final Host host, final boolean enabled) {
+ final OutOfBandManagement outOfBandManagementConfig = outOfBandManagementDao.findByHost(host.getId());
+ if (outOfBandManagementConfig == null) {
+ final OutOfBandManagementResponse response = new OutOfBandManagementResponse(null);
+ response.setSuccess(false);
+ response.setResultDescription("Out-of-band management is not configured for the host. Please configure the host before enabling/disabling it.");
+ return response;
+ }
+ hostAlertCache.invalidate(host.getId());
+ outOfBandManagementConfig.setEnabled(enabled);
+ boolean updateResult = outOfBandManagementDao.update(outOfBandManagementConfig.getId(), (OutOfBandManagementVO) outOfBandManagementConfig);
+ if (updateResult && !enabled) {
+ transitionPowerStateToDisabled(Collections.singletonList(host));
+ }
+ return buildEnableDisableResponse(enabled && updateResult);
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_CONFIGURE, eventDescription = "updating out-of-band management configuration")
+ public OutOfBandManagementResponse configureOutOfBandManagement(final Host host, final ImmutableMap<OutOfBandManagement.Option, String> options) {
+ OutOfBandManagement outOfBandManagementConfig = outOfBandManagementDao.findByHost(host.getId());
+ if (outOfBandManagementConfig == null) {
+ outOfBandManagementConfig = outOfBandManagementDao.persist(new OutOfBandManagementVO(host.getId()));
+ }
+ outOfBandManagementConfig = updateConfig(outOfBandManagementConfig, options);
+ if (Strings.isNullOrEmpty(outOfBandManagementConfig.getDriver()) || !outOfBandManagementDriversMap.containsKey(outOfBandManagementConfig.getDriver().toLowerCase())) {
+ throw new CloudRuntimeException("Out-of-band management driver is not available. Please provide a valid driver name.");
+ }
+
+ boolean updatedConfig = outOfBandManagementDao.update(outOfBandManagementConfig.getId(), (OutOfBandManagementVO) outOfBandManagementConfig);
+ CallContext.current().setEventDetails("host id:" + host.getId() + " configuration:" + outOfBandManagementConfig.getAddress() + ":" + outOfBandManagementConfig.getPort());
+
+ if (!updatedConfig) {
+ throw new CloudRuntimeException("Failed to update out-of-band management config for the host in the database.");
+ }
+
+ String result = "Out-of-band management successfully configured for the host";
+ LOG.debug(result);
+
+ final OutOfBandManagementResponse response = new OutOfBandManagementResponse(outOfBandManagementDao.findByHost(host.getId()));
+ response.setResultDescription(result);
+ response.setSuccess(true);
+ return response;
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_ACTION, eventDescription = "issuing Host out-of-band management action", async = true)
+ public OutOfBandManagementResponse executeOutOfBandManagementPowerOperation(final Host host, final OutOfBandManagement.PowerOperation powerOperation, final Long timeout) {
+ checkOutOfBandManagementEnabledByZoneClusterHost(host);
+ final OutOfBandManagement outOfBandManagementConfig = outOfBandManagementDao.findByHost(host.getId());
+ final ImmutableMap<OutOfBandManagement.Option, String> options = getOptions(outOfBandManagementConfig);
+ final OutOfBandManagementDriver driver = getDriver(outOfBandManagementConfig);
+
+ Long actionTimeOut = timeout;
+ if (actionTimeOut == null) {
+ actionTimeOut = OutOfBandManagementActionTimeout.valueIn(host.getClusterId());
+ }
+
+ final OutOfBandManagementDriverPowerCommand cmd = new OutOfBandManagementDriverPowerCommand(options, actionTimeOut, powerOperation);
+ final OutOfBandManagementDriverResponse driverResponse = driver.execute(cmd);
+
+ if (driverResponse == null) {
+ throw new CloudRuntimeException(String.format("Out-of-band Management action (%s) on host (%s) failed due to no response from the driver", powerOperation, host.getUuid()));
+ }
+
+ if (powerOperation.equals(OutOfBandManagement.PowerOperation.STATUS)) {
+ transitionPowerState(driverResponse.toEvent(), outOfBandManagementConfig);
+ }
+
+ if (!driverResponse.isSuccess()) {
+ String errorMessage = String.format("Out-of-band Management action (%s) on host (%s) failed with error: %s", powerOperation, host.getUuid(), driverResponse.getError());
+ if (driverResponse.hasAuthFailure()) {
+ errorMessage = String.format("Out-of-band Management action (%s) on host (%s) failed due to authentication error: %s. Please check configured credentials.", powerOperation, host.getUuid(), driverResponse.getError());
+ sendAuthError(host, errorMessage);
+ }
+ if (!powerOperation.equals(OutOfBandManagement.PowerOperation.STATUS)) {
+ LOG.debug(errorMessage);
+ }
+ throw new CloudRuntimeException(errorMessage);
+ }
+
+ final OutOfBandManagementResponse response = new OutOfBandManagementResponse(outOfBandManagementDao.findByHost(host.getId()));
+ response.setSuccess(driverResponse.isSuccess());
+ response.setResultDescription(driverResponse.getResult());
+ response.setId(host.getUuid());
+ response.setOutOfBandManagementAction(powerOperation.toString());
+ return response;
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_CHANGE_PASSWORD, eventDescription = "updating out-of-band management password")
+ public OutOfBandManagementResponse changeOutOfBandManagementPassword(final Host host, final String newPassword) {
+ checkOutOfBandManagementEnabledByZoneClusterHost(host);
+ if (Strings.isNullOrEmpty(newPassword)) {
+ throw new CloudRuntimeException(String.format("Cannot change out-of-band management password as provided new-password is null or empty for the host %s.", host.getUuid()));
+ }
+ GlobalLock outOfBandManagementHostLock = GlobalLock.getInternLock(getOutOfBandManagementHostLock(host.getId()));
+ try {
+ if (outOfBandManagementHostLock.lock(ACQUIRE_GLOBAL_LOCK_TIMEOUT_FOR_HOST)) {
+ try {
+ final OutOfBandManagement outOfBandManagementConfig = outOfBandManagementDao.findByHost(host.getId());
+
+ final ImmutableMap<OutOfBandManagement.Option, String> options = getOptions(outOfBandManagementConfig);
+ if (!(options.containsKey(OutOfBandManagement.Option.PASSWORD) && !Strings.isNullOrEmpty(options.get(OutOfBandManagement.Option.PASSWORD)))) {
+ throw new CloudRuntimeException(String.format("Cannot change out-of-band management password as we've no previously configured password for the host %s.", host.getUuid()));
+ }
+ final OutOfBandManagementDriver driver = getDriver(outOfBandManagementConfig);
+
+ final OutOfBandManagementDriverChangePasswordCommand cmd = new OutOfBandManagementDriverChangePasswordCommand(options, OutOfBandManagementActionTimeout.valueIn(host.getClusterId()), newPassword);
+ final OutOfBandManagementDriverResponse driverResponse;
+ try {
+ driverResponse = driver.execute(cmd);
+ } catch (Exception e) {
--- End diff --
Why are checked and non-checked exceptions being caught?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61847246
--- Diff: server/src/org/apache/cloudstack/outofbandmanagement/OutOfBandManagementServiceImpl.java ---
@@ -0,0 +1,532 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.outofbandmanagement;
+
+import com.cloud.alert.AlertManager;
+import com.cloud.dc.ClusterDetailsDao;
+import com.cloud.dc.ClusterDetailsVO;
+import com.cloud.dc.DataCenter;
+import com.cloud.dc.DataCenterDetailVO;
+import com.cloud.dc.dao.DataCenterDetailsDao;
+import com.cloud.domain.Domain;
+import com.cloud.event.ActionEvent;
+import com.cloud.event.ActionEventUtils;
+import com.cloud.event.EventTypes;
+import com.cloud.host.Host;
+import com.cloud.host.dao.HostDao;
+import com.cloud.org.Cluster;
+import com.cloud.utils.component.Manager;
+import com.cloud.utils.component.ManagerBase;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.db.Transaction;
+import com.cloud.utils.db.TransactionCallback;
+import com.cloud.utils.db.TransactionStatus;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.fsm.NoTransitionException;
+import com.google.common.base.Strings;
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import com.google.common.collect.ImmutableMap;
+import org.apache.cloudstack.api.response.OutOfBandManagementResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.framework.config.ConfigKey;
+import org.apache.cloudstack.framework.config.Configurable;
+import org.apache.cloudstack.outofbandmanagement.dao.OutOfBandManagementDao;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverChangePasswordCommand;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverPowerCommand;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverResponse;
+import org.apache.cloudstack.utils.identity.ManagementServerNode;
+import org.apache.log4j.Logger;
+import org.springframework.stereotype.Component;
+
+import javax.ejb.Local;
+import javax.inject.Inject;
+import javax.naming.ConfigurationException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+@Component
+@Local(value = {OutOfBandManagementService.class})
+public class OutOfBandManagementServiceImpl extends ManagerBase implements OutOfBandManagementService, Manager, Configurable {
+ public static final Logger LOG = Logger.getLogger(OutOfBandManagementServiceImpl.class);
+
+ @Inject
+ private ClusterDetailsDao clusterDetailsDao;
+ @Inject
+ private DataCenterDetailsDao dataCenterDetailsDao;
+ @Inject
+ private OutOfBandManagementDao outOfBandManagementDao;
+ @Inject
+ private HostDao hostDao;
+ @Inject
+ private AlertManager alertMgr;
+
+ private String name;
+ private long serviceId;
+
+ private List<OutOfBandManagementDriver> outOfBandManagementDrivers = new ArrayList<>();
+ private Map<String, OutOfBandManagementDriver> outOfBandManagementDriversMap = new HashMap<String, OutOfBandManagementDriver>();
+
+ private static final String OOBM_ENABLED_DETAIL = "outOfBandManagementEnabled";
+ private static final int ACQUIRE_GLOBAL_LOCK_TIMEOUT_FOR_HOST = 120;
+
+ private Cache<Long, Long> hostAlertCache;
+ private static ExecutorService backgroundSyncExecutor;
+
+ private String getOutOfBandManagementHostLock(long id) {
+ return "oobm.host." + id;
+ }
+
+ private void initializeDriversMap() {
+ if (outOfBandManagementDriversMap.isEmpty() && outOfBandManagementDrivers != null && outOfBandManagementDrivers.size() > 0) {
+ for (final OutOfBandManagementDriver driver : outOfBandManagementDrivers) {
+ outOfBandManagementDriversMap.put(driver.getName().toLowerCase(), driver);
+ }
+ LOG.debug("Discovered out-of-band management drivers configured in the OutOfBandManagementService");
+ }
+ }
+
+ private OutOfBandManagementDriver getDriver(final OutOfBandManagement outOfBandManagementConfig) {
+ if (!Strings.isNullOrEmpty(outOfBandManagementConfig.getDriver())) {
+ final OutOfBandManagementDriver driver = outOfBandManagementDriversMap.get(outOfBandManagementConfig.getDriver());
+ if (driver != null) {
+ return driver;
+ }
+ }
+ throw new CloudRuntimeException("Configured out-of-band management driver is not available. Aborting any out-of-band management action.");
+ }
+
+ protected OutOfBandManagement updateConfig(final OutOfBandManagement outOfBandManagementConfig, final ImmutableMap<OutOfBandManagement.Option, String> options) {
+ if (outOfBandManagementConfig == null) {
+ throw new CloudRuntimeException("Out-of-band management is not configured for the host. Aborting.");
+ }
+ if (options == null) {
+ return outOfBandManagementConfig;
+ }
+ for (OutOfBandManagement.Option option: options.keySet()) {
+ final String value = options.get(option);
+ if (Strings.isNullOrEmpty(value)) {
+ continue;
+ }
+ switch (option) {
+ case DRIVER:
+ outOfBandManagementConfig.setDriver(value);
+ break;
+ case ADDRESS:
+ outOfBandManagementConfig.setAddress(value);
+ break;
+ case PORT:
+ outOfBandManagementConfig.setPort(Integer.parseInt(value));
+ break;
+ case USERNAME:
+ outOfBandManagementConfig.setUsername(value);
+ break;
+ case PASSWORD:
+ outOfBandManagementConfig.setPassword(value);
+ break;
+ }
+ }
+ return outOfBandManagementConfig;
+ }
+
+ protected ImmutableMap<OutOfBandManagement.Option, String> getOptions(final OutOfBandManagement outOfBandManagementConfig) {
+ final ImmutableMap.Builder<OutOfBandManagement.Option, String> optionsBuilder = ImmutableMap.builder();
+ if (outOfBandManagementConfig == null) {
+ throw new CloudRuntimeException("Out-of-band management is not configured for the host. Aborting.");
+ }
+ for (OutOfBandManagement.Option option: OutOfBandManagement.Option.values()) {
+ String value = null;
+ switch (option) {
+ case DRIVER:
+ value = outOfBandManagementConfig.getDriver();
+ break;
+ case ADDRESS:
+ value = outOfBandManagementConfig.getAddress();
+ break;
+ case PORT:
+ if (outOfBandManagementConfig.getPort() != null) {
+ value = String.valueOf(outOfBandManagementConfig.getPort());
+ }
+ break;
+ case USERNAME:
+ value = outOfBandManagementConfig.getUsername();
+ break;
+ case PASSWORD:
+ value = outOfBandManagementConfig.getPassword();
+ break;
+ }
+ if (value != null) {
+ optionsBuilder.put(option, value);
+ }
+ }
+ return optionsBuilder.build();
+ }
+
+ private void sendAuthError(final Host host, final String message) {
+ try {
+ hostAlertCache.asMap().putIfAbsent(host.getId(), 0L);
+ Long sentCount = hostAlertCache.asMap().get(host.getId());
+ if (sentCount != null && sentCount <= 0) {
+ boolean concurrentUpdateResult = hostAlertCache.asMap().replace(host.getId(), sentCount, sentCount+1L);
+ if (concurrentUpdateResult) {
+ final String subject = String.format("Out-of-band management auth-error detected for host:%d in cluster:%d, zone:%d", host.getId(), host.getClusterId(), host.getDataCenterId());
+ LOG.error(subject + ": " + message);
+ alertMgr.sendAlert(AlertManager.AlertType.ALERT_TYPE_OOBM_AUTH_ERROR, host.getDataCenterId(), host.getPodId(), subject, message);
+ }
+ }
+ } catch (Exception ignored) {
+ }
+ }
+
+ private boolean transitionPowerState(OutOfBandManagement.PowerState.Event event, OutOfBandManagement outOfBandManagementHost) {
+ if (outOfBandManagementHost == null) {
+ return false;
+ }
+ OutOfBandManagement.PowerState currentPowerState = outOfBandManagementHost.getPowerState();
+ try {
+ OutOfBandManagement.PowerState newPowerState = OutOfBandManagement.PowerState.getStateMachine().getNextState(currentPowerState, event);
+ boolean result = outOfBandManagementDao.updateState(currentPowerState, event, newPowerState, outOfBandManagementHost, null);
+ if (result) {
+ final String message = String.format("Transitioned out-of-band management power state from:%s to:%s due to event:%s for the host id:%d", currentPowerState, newPowerState, event, outOfBandManagementHost.getHostId());
+ LOG.debug(message);
+ ActionEventUtils.onActionEvent(CallContext.current().getCallingUserId(), CallContext.current().getCallingAccountId(), Domain.ROOT_DOMAIN,
+ EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_POWERSTATE_TRANSITION, message);
+ }
+ return result;
+ } catch (NoTransitionException ignored) {
+ LOG.trace(String.format("Unable to transition out-of-band management power state for host id=%s for the event=%s and current power state=%s", outOfBandManagementHost.getHostId(), event, currentPowerState));
+ }
+ return false;
+ }
+
+ private boolean isOutOfBandManagementEnabledForZone(Long zoneId) {
+ if (zoneId == null) {
+ return true;
+ }
+ final DataCenterDetailVO zoneDetails = dataCenterDetailsDao.findDetail(zoneId, OOBM_ENABLED_DETAIL);
+ if (zoneDetails != null && !Strings.isNullOrEmpty(zoneDetails.getValue()) && !Boolean.valueOf(zoneDetails.getValue())) {
+ return false;
+ }
+ return true;
+ }
+
+ private boolean isOutOfBandManagementEnabledForCluster(Long clusterId) {
+ if (clusterId == null) {
+ return true;
+ }
+ final ClusterDetailsVO clusterDetails = clusterDetailsDao.findDetail(clusterId, OOBM_ENABLED_DETAIL);
+ if (clusterDetails != null && !Strings.isNullOrEmpty(clusterDetails.getValue()) && !Boolean.valueOf(clusterDetails.getValue())) {
+ return false;
+ }
+ return true;
+ }
+
+ private boolean isOutOfBandManagementEnabledForHost(Long hostId) {
+ if (hostId == null) {
+ return false;
+ }
+ final OutOfBandManagement outOfBandManagementConfig = outOfBandManagementDao.findByHost(hostId);
+ if (outOfBandManagementConfig == null || !outOfBandManagementConfig.isEnabled()) {
+ return false;
+ }
+ return true;
+ }
+
+ private void checkOutOfBandManagementEnabledByZoneClusterHost(final Host host) {
+ if (!isOutOfBandManagementEnabledForZone(host.getDataCenterId())) {
+ throw new CloudRuntimeException("Out-of-band management is disabled for the host's zone. Aborting Operation.");
+ }
+ if (!isOutOfBandManagementEnabledForCluster(host.getClusterId())) {
+ throw new CloudRuntimeException("Out-of-band management is disabled for the host's cluster. Aborting Operation.");
+ }
+ if (!isOutOfBandManagementEnabledForHost(host.getId())) {
+ throw new CloudRuntimeException("Out-of-band management is disabled or not configured for the host. Aborting Operation.");
+ }
+ }
+
+ public boolean isOutOfBandManagementEnabled(final Host host) {
+ return isOutOfBandManagementEnabledForZone(host.getDataCenterId())
+ && isOutOfBandManagementEnabledForCluster(host.getClusterId())
+ && isOutOfBandManagementEnabledForHost(host.getId());
+ }
+
+ public boolean transitionPowerStateToDisabled(List<? extends Host> hosts) {
+ boolean result = true;
+ for (Host host : hosts) {
+ result = result && transitionPowerState(OutOfBandManagement.PowerState.Event.Disabled,
+ outOfBandManagementDao.findByHost(host.getId()));
+ }
+ return result;
+ }
+
+ public void submitBackgroundPowerSyncTask(final Host host) {
+ if (host != null) {
+ // Note: This is a blocking queue based executor
+ backgroundSyncExecutor.submit(new OutOfBandManagementBackgroundTask(this, host, OutOfBandManagement.PowerOperation.STATUS));
+ }
+ }
+
+ private OutOfBandManagementResponse buildEnableDisableResponse(final boolean enabled) {
+ final OutOfBandManagementResponse response = new OutOfBandManagementResponse();
+ response.setEnabled(enabled);
+ response.setSuccess(true);
+ return response;
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_ENABLEDISABLE, eventDescription = "enabling/disabling out-of-band management on a zone")
+ public OutOfBandManagementResponse enableDisableOutOfBandManagement(final DataCenter zone, final boolean enabled) {
+ dataCenterDetailsDao.persist(zone.getId(), OOBM_ENABLED_DETAIL, String.valueOf(enabled));
+ if (!enabled) {
+ transitionPowerStateToDisabled(hostDao.findByDataCenterId(zone.getId()));
+ }
+ return buildEnableDisableResponse(enabled);
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_ENABLEDISABLE, eventDescription = "enabling/disabling out-of-band management on a cluster")
+ public OutOfBandManagementResponse enableDisableOutOfBandManagement(final Cluster cluster, final boolean enabled) {
+ clusterDetailsDao.persist(cluster.getId(), OOBM_ENABLED_DETAIL, String.valueOf(enabled));
+ if (!enabled) {
+ transitionPowerStateToDisabled(hostDao.findByClusterId(cluster.getId()));
+ }
+ return buildEnableDisableResponse(enabled);
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_ENABLEDISABLE, eventDescription = "enabling/disabling out-of-band management on a host")
+ public OutOfBandManagementResponse enableDisableOutOfBandManagement(final Host host, final boolean enabled) {
+ final OutOfBandManagement outOfBandManagementConfig = outOfBandManagementDao.findByHost(host.getId());
+ if (outOfBandManagementConfig == null) {
+ final OutOfBandManagementResponse response = new OutOfBandManagementResponse(null);
+ response.setSuccess(false);
+ response.setResultDescription("Out-of-band management is not configured for the host. Please configure the host before enabling/disabling it.");
+ return response;
+ }
+ hostAlertCache.invalidate(host.getId());
+ outOfBandManagementConfig.setEnabled(enabled);
+ boolean updateResult = outOfBandManagementDao.update(outOfBandManagementConfig.getId(), (OutOfBandManagementVO) outOfBandManagementConfig);
+ if (updateResult && !enabled) {
+ transitionPowerStateToDisabled(Collections.singletonList(host));
+ }
+ return buildEnableDisableResponse(enabled && updateResult);
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_CONFIGURE, eventDescription = "updating out-of-band management configuration")
+ public OutOfBandManagementResponse configureOutOfBandManagement(final Host host, final ImmutableMap<OutOfBandManagement.Option, String> options) {
+ OutOfBandManagement outOfBandManagementConfig = outOfBandManagementDao.findByHost(host.getId());
+ if (outOfBandManagementConfig == null) {
+ outOfBandManagementConfig = outOfBandManagementDao.persist(new OutOfBandManagementVO(host.getId()));
+ }
+ outOfBandManagementConfig = updateConfig(outOfBandManagementConfig, options);
+ if (Strings.isNullOrEmpty(outOfBandManagementConfig.getDriver()) || !outOfBandManagementDriversMap.containsKey(outOfBandManagementConfig.getDriver().toLowerCase())) {
+ throw new CloudRuntimeException("Out-of-band management driver is not available. Please provide a valid driver name.");
+ }
+
+ boolean updatedConfig = outOfBandManagementDao.update(outOfBandManagementConfig.getId(), (OutOfBandManagementVO) outOfBandManagementConfig);
+ CallContext.current().setEventDetails("host id:" + host.getId() + " configuration:" + outOfBandManagementConfig.getAddress() + ":" + outOfBandManagementConfig.getPort());
+
+ if (!updatedConfig) {
+ throw new CloudRuntimeException("Failed to update out-of-band management config for the host in the database.");
+ }
+
+ String result = "Out-of-band management successfully configured for the host";
+ LOG.debug(result);
+
+ final OutOfBandManagementResponse response = new OutOfBandManagementResponse(outOfBandManagementDao.findByHost(host.getId()));
+ response.setResultDescription(result);
+ response.setSuccess(true);
+ return response;
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_ACTION, eventDescription = "issuing Host out-of-band management action", async = true)
+ public OutOfBandManagementResponse executeOutOfBandManagementPowerOperation(final Host host, final OutOfBandManagement.PowerOperation powerOperation, final Long timeout) {
+ checkOutOfBandManagementEnabledByZoneClusterHost(host);
+ final OutOfBandManagement outOfBandManagementConfig = outOfBandManagementDao.findByHost(host.getId());
+ final ImmutableMap<OutOfBandManagement.Option, String> options = getOptions(outOfBandManagementConfig);
+ final OutOfBandManagementDriver driver = getDriver(outOfBandManagementConfig);
+
+ Long actionTimeOut = timeout;
+ if (actionTimeOut == null) {
+ actionTimeOut = OutOfBandManagementActionTimeout.valueIn(host.getClusterId());
+ }
+
+ final OutOfBandManagementDriverPowerCommand cmd = new OutOfBandManagementDriverPowerCommand(options, actionTimeOut, powerOperation);
+ final OutOfBandManagementDriverResponse driverResponse = driver.execute(cmd);
+
+ if (driverResponse == null) {
+ throw new CloudRuntimeException(String.format("Out-of-band Management action (%s) on host (%s) failed due to no response from the driver", powerOperation, host.getUuid()));
+ }
+
+ if (powerOperation.equals(OutOfBandManagement.PowerOperation.STATUS)) {
+ transitionPowerState(driverResponse.toEvent(), outOfBandManagementConfig);
+ }
+
+ if (!driverResponse.isSuccess()) {
+ String errorMessage = String.format("Out-of-band Management action (%s) on host (%s) failed with error: %s", powerOperation, host.getUuid(), driverResponse.getError());
+ if (driverResponse.hasAuthFailure()) {
+ errorMessage = String.format("Out-of-band Management action (%s) on host (%s) failed due to authentication error: %s. Please check configured credentials.", powerOperation, host.getUuid(), driverResponse.getError());
+ sendAuthError(host, errorMessage);
+ }
+ if (!powerOperation.equals(OutOfBandManagement.PowerOperation.STATUS)) {
+ LOG.debug(errorMessage);
+ }
+ throw new CloudRuntimeException(errorMessage);
+ }
+
+ final OutOfBandManagementResponse response = new OutOfBandManagementResponse(outOfBandManagementDao.findByHost(host.getId()));
+ response.setSuccess(driverResponse.isSuccess());
+ response.setResultDescription(driverResponse.getResult());
+ response.setId(host.getUuid());
+ response.setOutOfBandManagementAction(powerOperation.toString());
+ return response;
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_CHANGE_PASSWORD, eventDescription = "updating out-of-band management password")
+ public OutOfBandManagementResponse changeOutOfBandManagementPassword(final Host host, final String newPassword) {
+ checkOutOfBandManagementEnabledByZoneClusterHost(host);
+ if (Strings.isNullOrEmpty(newPassword)) {
+ throw new CloudRuntimeException(String.format("Cannot change out-of-band management password as provided new-password is null or empty for the host %s.", host.getUuid()));
+ }
+ GlobalLock outOfBandManagementHostLock = GlobalLock.getInternLock(getOutOfBandManagementHostLock(host.getId()));
+ try {
+ if (outOfBandManagementHostLock.lock(ACQUIRE_GLOBAL_LOCK_TIMEOUT_FOR_HOST)) {
+ try {
--- End diff --
we want to handle lock failures separately than critical code failure
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61793974
--- Diff: server/src/com/cloud/server/StatsCollector.java ---
@@ -251,8 +262,9 @@ public boolean start() {
}
private void init(Map<String, String> configs) {
- _executor = Executors.newScheduledThreadPool(4, new NamedThreadFactory("StatsCollector"));
+ _executor = Executors.newScheduledThreadPool(6, new NamedThreadFactory("StatsCollector"));
--- End diff --
``6`` seems like a magic value. How was this value determined?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-216229281
PR is code complete with testing efforts that @borisstoyanov can share
tag:mergeready
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61777306
--- Diff: api/src/org/apache/cloudstack/api/command/admin/outofbandmanagement/IssueOutOfBandManagementPowerActionCmd.java ---
@@ -0,0 +1,128 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.api.command.admin.outofbandmanagement;
+
+import com.cloud.event.EventTypes;
+import com.cloud.exception.ConcurrentOperationException;
+import com.cloud.exception.InsufficientCapacityException;
+import com.cloud.exception.NetworkRuleConflictException;
+import com.cloud.exception.ResourceAllocationException;
+import com.cloud.exception.ResourceUnavailableException;
+import com.cloud.host.Host;
+import com.google.common.base.Strings;
+import org.apache.cloudstack.acl.RoleType;
+import org.apache.cloudstack.api.APICommand;
+import org.apache.cloudstack.api.ApiCommandJobType;
+import org.apache.cloudstack.api.ApiConstants;
+import org.apache.cloudstack.api.ApiErrorCode;
+import org.apache.cloudstack.api.BaseAsyncCmd;
+import org.apache.cloudstack.api.Parameter;
+import org.apache.cloudstack.api.ServerApiException;
+import org.apache.cloudstack.api.response.HostResponse;
+import org.apache.cloudstack.api.response.OutOfBandManagementResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagement.PowerOperation;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagementService;
+
+import javax.inject.Inject;
+
+@APICommand(name = "issueOutOfBandManagementPowerAction", description = "Initiates the specified power action to the host's out-of-band management interface",
--- End diff --
Please add the version to this annotation
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61776973
--- Diff: api/src/org/apache/cloudstack/api/command/admin/outofbandmanagement/DisableOutOfBandManagementForHostCmd.java ---
@@ -0,0 +1,107 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.api.command.admin.outofbandmanagement;
+
+import com.cloud.event.EventTypes;
+import com.cloud.exception.ConcurrentOperationException;
+import com.cloud.exception.InsufficientCapacityException;
+import com.cloud.exception.NetworkRuleConflictException;
+import com.cloud.exception.ResourceAllocationException;
+import com.cloud.exception.ResourceUnavailableException;
+import com.cloud.host.Host;
+import org.apache.cloudstack.acl.RoleType;
+import org.apache.cloudstack.api.APICommand;
+import org.apache.cloudstack.api.ApiConstants;
+import org.apache.cloudstack.api.ApiErrorCode;
+import org.apache.cloudstack.api.BaseAsyncCmd;
+import org.apache.cloudstack.api.BaseCmd;
+import org.apache.cloudstack.api.Parameter;
+import org.apache.cloudstack.api.ServerApiException;
+import org.apache.cloudstack.api.response.HostResponse;
+import org.apache.cloudstack.api.response.OutOfBandManagementResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagementService;
+
+import javax.inject.Inject;
+
+@APICommand(name = "disableOutOfBandManagementForHost", description = "Disables out-of-band management for a host",
--- End diff --
Please add the version to this annotation
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61814182
--- Diff: server/test/org/apache/cloudstack/outofbandmanagement/OutOfBandManagementServiceTest.java ---
@@ -0,0 +1,119 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.outofbandmanagement;
+
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.google.common.collect.ImmutableMap;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverResponse;
+import org.junit.Assert;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.mockito.runners.MockitoJUnitRunner;
+
+@RunWith(MockitoJUnitRunner.class)
+public class OutOfBandManagementServiceTest {
+
+ OutOfBandManagementServiceImpl oobmService = new OutOfBandManagementServiceImpl();
+
+ @Test
+ public void testOutOfBandManagementDriverResponseEvent() {
+ OutOfBandManagementDriverResponse r = new OutOfBandManagementDriverResponse("some result", "some error", false);
+
+ r.setSuccess(false);
+ r.setAuthFailure(false);
+ Assert.assertEquals(r.toEvent(), OutOfBandManagement.PowerState.Event.Unknown);
+
+ r.setSuccess(false);
+ r.setAuthFailure(true);
+ Assert.assertEquals(r.toEvent(), OutOfBandManagement.PowerState.Event.AuthError);
+
+ r.setAuthFailure(false);
+ r.setSuccess(true);
+ r.setPowerState(OutOfBandManagement.PowerState.On);
+ Assert.assertEquals(r.toEvent(), OutOfBandManagement.PowerState.Event.On);
+
+ r.setPowerState(OutOfBandManagement.PowerState.Off);
+ Assert.assertEquals(r.toEvent(), OutOfBandManagement.PowerState.Event.Off);
+
+ r.setPowerState(OutOfBandManagement.PowerState.Disabled);
+ Assert.assertEquals(r.toEvent(), OutOfBandManagement.PowerState.Event.Disabled);
+ }
+
+ private ImmutableMap<OutOfBandManagement.Option, String> buildRandomOptionsMap() {
+ ImmutableMap.Builder<OutOfBandManagement.Option, String> builder = new ImmutableMap.Builder<>();
+ builder.put(OutOfBandManagement.Option.ADDRESS, "localhost");
+ builder.put(OutOfBandManagement.Option.DRIVER, "ipmitool");
+ return builder.build();
+ }
+
+ @Test
+ public void testUpdateOutOfBandManagementConfigValid() {
+ OutOfBandManagement config = new OutOfBandManagementVO(123L);
+ Assert.assertEquals(config.getPowerState(), OutOfBandManagement.PowerState.Disabled);
+ config = oobmService.updateConfig(config, buildRandomOptionsMap());
+ Assert.assertEquals(config.getAddress(), "localhost");
+ Assert.assertEquals(config.getDriver(), "ipmitool");
+ Assert.assertEquals(config.getPowerState(), OutOfBandManagement.PowerState.Disabled);
+ }
+
+ @Test
+ public void testUpdateOutOfBandManagementConfigInValid() {
+ try {
+ oobmService.updateConfig(null, buildRandomOptionsMap());
+ Assert.fail("CloudRuntimeException was expect for out-of-band management not configured for the host");
+ } catch (CloudRuntimeException ignored) {
+ }
+
+ try {
+ oobmService.updateConfig(null, null);
+ Assert.fail("CloudRuntimeException was expect for out-of-band management not configured for the host");
+ } catch (CloudRuntimeException ignored) {
+ }
+
+ OutOfBandManagement config = new OutOfBandManagementVO(123L);
+ config.setAddress(null);
+ config = oobmService.updateConfig(config, null);
+ Assert.assertEquals(config.getAddress(), null);
+ Assert.assertEquals(config.getPowerState(), OutOfBandManagement.PowerState.Disabled);
--- End diff --
Consider dividing this test method into three -- one for each failure scenario and another for the success scenario. Also, for the failure methods, use the ``expected`` attribute on the ``@Test`` annotation.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61844463
--- Diff: api/src/org/apache/cloudstack/outofbandmanagement/OutOfBandManagementService.java ---
@@ -0,0 +1,51 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.outofbandmanagement;
+
+import com.cloud.dc.DataCenter;
+import com.cloud.host.Host;
+import com.cloud.org.Cluster;
+import com.google.common.collect.ImmutableMap;
+import org.apache.cloudstack.api.response.OutOfBandManagementResponse;
+import org.apache.cloudstack.framework.config.ConfigKey;
+
+import java.util.List;
+
+public interface OutOfBandManagementService {
+
+ ConfigKey<Long> OutOfBandManagementActionTimeout = new ConfigKey<Long>("Advanced", Long.class, "outofbandmanagement.action.timeout", "60",
+ "The out of band management action timeout in seconds, configurable by cluster", true, ConfigKey.Scope.Cluster);
+
+ ConfigKey<Long> OutOfBandManagementSyncThreadInterval = new ConfigKey<Long>("Advanced", Long.class, "outofbandmanagement.sync.interval", "300000",
+ "The interval (in milliseconds) when the out-of-band management background sync are retrieved", true, ConfigKey.Scope.Global);
+
+ ConfigKey<Integer> OutOfBandManagementSyncThreadPoolSize = new ConfigKey<Integer>("Advanced", Integer.class, "outofbandmanagement.sync.poolsize", "50",
+ "The out of band management background sync thread pool size", true, ConfigKey.Scope.Global);
+
+ long getId();
+ boolean isOutOfBandManagementEnabled(Host host);
+ void submitBackgroundPowerSyncTask(Host host);
+ boolean transitionPowerStateToDisabled(List<? extends Host> hosts);
+
+ OutOfBandManagementResponse enableDisableOutOfBandManagement(DataCenter zone, boolean enabled);
+ OutOfBandManagementResponse enableDisableOutOfBandManagement(Cluster cluster, boolean enabled);
+ OutOfBandManagementResponse enableDisableOutOfBandManagement(Host host, boolean enabled);
--- End diff --
Fixed, methods are separated now
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61847561
--- Diff: server/test/org/apache/cloudstack/outofbandmanagement/OutOfBandManagementServiceTest.java ---
@@ -0,0 +1,119 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.outofbandmanagement;
+
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.google.common.collect.ImmutableMap;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverResponse;
+import org.junit.Assert;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.mockito.runners.MockitoJUnitRunner;
+
+@RunWith(MockitoJUnitRunner.class)
+public class OutOfBandManagementServiceTest {
+
+ OutOfBandManagementServiceImpl oobmService = new OutOfBandManagementServiceImpl();
+
+ @Test
+ public void testOutOfBandManagementDriverResponseEvent() {
+ OutOfBandManagementDriverResponse r = new OutOfBandManagementDriverResponse("some result", "some error", false);
+
+ r.setSuccess(false);
+ r.setAuthFailure(false);
+ Assert.assertEquals(r.toEvent(), OutOfBandManagement.PowerState.Event.Unknown);
+
+ r.setSuccess(false);
+ r.setAuthFailure(true);
+ Assert.assertEquals(r.toEvent(), OutOfBandManagement.PowerState.Event.AuthError);
+
+ r.setAuthFailure(false);
+ r.setSuccess(true);
+ r.setPowerState(OutOfBandManagement.PowerState.On);
+ Assert.assertEquals(r.toEvent(), OutOfBandManagement.PowerState.Event.On);
+
+ r.setPowerState(OutOfBandManagement.PowerState.Off);
+ Assert.assertEquals(r.toEvent(), OutOfBandManagement.PowerState.Event.Off);
+
+ r.setPowerState(OutOfBandManagement.PowerState.Disabled);
+ Assert.assertEquals(r.toEvent(), OutOfBandManagement.PowerState.Event.Disabled);
+ }
+
+ private ImmutableMap<OutOfBandManagement.Option, String> buildRandomOptionsMap() {
+ ImmutableMap.Builder<OutOfBandManagement.Option, String> builder = new ImmutableMap.Builder<>();
+ builder.put(OutOfBandManagement.Option.ADDRESS, "localhost");
+ builder.put(OutOfBandManagement.Option.DRIVER, "ipmitool");
+ return builder.build();
+ }
+
+ @Test
+ public void testUpdateOutOfBandManagementConfigValid() {
+ OutOfBandManagement config = new OutOfBandManagementVO(123L);
+ Assert.assertEquals(config.getPowerState(), OutOfBandManagement.PowerState.Disabled);
+ config = oobmService.updateConfig(config, buildRandomOptionsMap());
+ Assert.assertEquals(config.getAddress(), "localhost");
+ Assert.assertEquals(config.getDriver(), "ipmitool");
+ Assert.assertEquals(config.getPowerState(), OutOfBandManagement.PowerState.Disabled);
+ }
+
+ @Test
+ public void testUpdateOutOfBandManagementConfigInValid() {
+ try {
+ oobmService.updateConfig(null, buildRandomOptionsMap());
+ Assert.fail("CloudRuntimeException was expect for out-of-band management not configured for the host");
+ } catch (CloudRuntimeException ignored) {
+ }
+
+ try {
+ oobmService.updateConfig(null, null);
+ Assert.fail("CloudRuntimeException was expect for out-of-band management not configured for the host");
+ } catch (CloudRuntimeException ignored) {
+ }
+
+ OutOfBandManagement config = new OutOfBandManagementVO(123L);
+ config.setAddress(null);
+ config = oobmService.updateConfig(config, null);
+ Assert.assertEquals(config.getAddress(), null);
+ Assert.assertEquals(config.getPowerState(), OutOfBandManagement.PowerState.Disabled);
+ }
+
+ @Test
+ public void testGetOutOfBandManagementOptionsValid() {
+ OutOfBandManagement configEmpty = new OutOfBandManagementVO(123L);
+ ImmutableMap<OutOfBandManagement.Option, String> optionsEmpty = oobmService.getOptions(configEmpty);
+ Assert.assertEquals(optionsEmpty.size(), 0);
+
+ OutOfBandManagement config = new OutOfBandManagementVO(123L);
+ config.setAddress("localhost");
+ config.setDriver("ipmitool");
+ config.setPort(1234);
+ ImmutableMap<OutOfBandManagement.Option, String> options = oobmService.getOptions(config);
+ Assert.assertEquals(options.get(OutOfBandManagement.Option.ADDRESS), "localhost");
+ Assert.assertEquals(options.get(OutOfBandManagement.Option.DRIVER), "ipmitool");
+ Assert.assertEquals(options.get(OutOfBandManagement.Option.PORT), "1234");
+ }
+
+ @Test
+ public void testGetOutOfBandManagementOptionsInvalid() {
+ try {
+ oobmService.getOptions(null);
+ Assert.fail("CloudRuntimeException was expected for finding options of host with out-of-band management configuration");
+ } catch (CloudRuntimeException e) {
+ }
--- End diff --
Fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by swill <gi...@git.apache.org>.
Github user swill commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-217515255
Both this one and dynamic roles are queued up for CI as soon as I stabilize master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-216908315
For the issue reported by Travis, I force it to install ipmitool version 1.8.16; but looks like even that version causes the same failure (some times).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-217501592
@jburwell thanks, I'll sort them out. Meanwhile, if we can get the dynamic-roles PR merged today/tomorrow, I can use the annotations/validation usage in APIs and further improve the PR
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r62616838
--- Diff: utils/src/main/java/org/apache/cloudstack/utils/process/ProcessResult.java ---
@@ -0,0 +1,46 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.utils.process;
+
+public class ProcessResult {
--- End diff --
Not making this nested inside ProcessRunner, ProcessResult is used in oobm driver. The constructor is the only way to initialize the inner items, which are final. The class if final now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r62444371
--- Diff: utils/src/main/java/org/apache/cloudstack/utils/process/ProcessRunner.java ---
@@ -0,0 +1,111 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+
+package org.apache.cloudstack.utils.process;
+
+import org.apache.log4j.Logger;
+
+import java.io.IOException;
+import java.util.List;
+
+public class ProcessRunner {
+ public static final Logger LOG = Logger.getLogger(ProcessRunner.class);
+
+ private String stdOutput;
+ private String stdError;
+ private int returnCode = -1;
+
+ public String getStdOutput() {
+ return stdOutput;
+ }
+
+ public void setStdOutput(String stdOutput) {
+ this.stdOutput = stdOutput;
+ }
+
+ public String getStdError() {
+ return stdError;
+ }
+
+ public void setStdError(String stdError) {
+ this.stdError = stdError;
+ }
+
+ public int getReturnCode() {
+ return returnCode;
+ }
+
+ public void setReturnCode(int returnCode) {
+ this.returnCode = returnCode;
+ }
+
+ public static ProcessRunner executeCommands(List<String> commands, long timeOutSeconds) {
+ ProcessRunner result = new ProcessRunner();
+ try {
+ Process process = new ProcessBuilder().command(commands).start();
+ StreamGobbler stdInputGobbler = new StreamGobbler(process.getInputStream());
+ StreamGobbler stdErrorGobbler = new StreamGobbler(process.getErrorStream());
+ stdInputGobbler.start();
+ stdErrorGobbler.start();
+
+ if (timeOutSeconds > 0) {
+ ProcessWaitForThread worker = new ProcessWaitForThread(process);
--- End diff --
Removed use of this idiom, instead using Future and .get(timeout, ...).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by borisstoyanov <gi...@git.apache.org>.
Github user borisstoyanov commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-217414515
I've just run final sanity tests on this PR, all the OOBM features are working and did not observed any side effects, so I'm happy with the latest changes getting merged.
LGTM!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r62790877
--- Diff: utils/src/main/java/org/apache/cloudstack/utils/process/ProcessRunner.java ---
@@ -0,0 +1,112 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+
+package org.apache.cloudstack.utils.process;
+
+import com.cloud.utils.concurrency.NamedThreadFactory;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Strings;
+import org.apache.log4j.Logger;
+import org.joda.time.Duration;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.List;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+public class ProcessRunner {
+ public static final Logger LOG = Logger.getLogger(ProcessRunner.class);
+
+ private static final ExecutorService processExecutor = Executors.newCachedThreadPool(new NamedThreadFactory("ProcessRunner"));
+
+ private static String readStream(final InputStream inputStream) {
+ String text = null;
+ try {
+ final BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(inputStream));
+ String line;
+ while ((line = bufferedReader.readLine()) != null) {
+ if (Strings.isNullOrEmpty(text)) {
+ text = line;
+ } else {
+ text = text + "\n" + line;
+ }
+ }
+ } catch (IOException e) {
+ if (LOG.isTraceEnabled()) {
+ LOG.trace("ProcessRunner::readStream failed due to: " + e.getMessage());
+ }
+ }
+ return text;
+ }
+
+ public static ProcessResult executeCommands(final List<String> commands, final Duration timeOut) {
+ Preconditions.checkArgument(commands != null && timeOut != null);
+
+ int retVal = -2;
+ String stdOutput = null;
+ String stdError = null;
+
+ try {
+ final Process process = new ProcessBuilder().command(commands).start();
+ if (timeOut.getStandardSeconds() > 0) {
--- End diff --
@jburwell ipmitool driver does not execute the command with a timeout, the utility needs to be generic which is why an option is provided to no consider timeout (0 = no timeout). The users of the utility need to make the decision and not this utility. For certain cases, it may be acceptable to execute a process without any time boundary therefore the utility should allow for both use-cases.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by swill <gi...@git.apache.org>.
Github user swill commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-217319127
OK. Sounds good. :) Thx...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by bhaisaab <gi...@git.apache.org>.
Github user bhaisaab commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-213520037
Thanks @swill @kiwiflyer
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by swill <gi...@git.apache.org>.
Github user swill commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-214796529
@rhtyd not really related to this PR, but a question I have. How should we be tracking environment change requirements? Ideally we would have a mechanism to flag this type of change. I understand it is only needed for the tests to pass, so it won't be a blocker for people just using it, but I am wondering how we should be handling this stuff in general. Do you have ideas? Maybe we can try to call it out in the initial PR or something like that? Ideally this info would show up in the release notes as well. What do you think @pdion891?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61803751
--- Diff: server/src/org/apache/cloudstack/outofbandmanagement/OutOfBandManagementServiceImpl.java ---
@@ -0,0 +1,532 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.outofbandmanagement;
+
+import com.cloud.alert.AlertManager;
+import com.cloud.dc.ClusterDetailsDao;
+import com.cloud.dc.ClusterDetailsVO;
+import com.cloud.dc.DataCenter;
+import com.cloud.dc.DataCenterDetailVO;
+import com.cloud.dc.dao.DataCenterDetailsDao;
+import com.cloud.domain.Domain;
+import com.cloud.event.ActionEvent;
+import com.cloud.event.ActionEventUtils;
+import com.cloud.event.EventTypes;
+import com.cloud.host.Host;
+import com.cloud.host.dao.HostDao;
+import com.cloud.org.Cluster;
+import com.cloud.utils.component.Manager;
+import com.cloud.utils.component.ManagerBase;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.db.Transaction;
+import com.cloud.utils.db.TransactionCallback;
+import com.cloud.utils.db.TransactionStatus;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.fsm.NoTransitionException;
+import com.google.common.base.Strings;
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import com.google.common.collect.ImmutableMap;
+import org.apache.cloudstack.api.response.OutOfBandManagementResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.framework.config.ConfigKey;
+import org.apache.cloudstack.framework.config.Configurable;
+import org.apache.cloudstack.outofbandmanagement.dao.OutOfBandManagementDao;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverChangePasswordCommand;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverPowerCommand;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverResponse;
+import org.apache.cloudstack.utils.identity.ManagementServerNode;
+import org.apache.log4j.Logger;
+import org.springframework.stereotype.Component;
+
+import javax.ejb.Local;
+import javax.inject.Inject;
+import javax.naming.ConfigurationException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+@Component
+@Local(value = {OutOfBandManagementService.class})
+public class OutOfBandManagementServiceImpl extends ManagerBase implements OutOfBandManagementService, Manager, Configurable {
+ public static final Logger LOG = Logger.getLogger(OutOfBandManagementServiceImpl.class);
+
+ @Inject
+ private ClusterDetailsDao clusterDetailsDao;
+ @Inject
+ private DataCenterDetailsDao dataCenterDetailsDao;
+ @Inject
+ private OutOfBandManagementDao outOfBandManagementDao;
+ @Inject
+ private HostDao hostDao;
+ @Inject
+ private AlertManager alertMgr;
+
+ private String name;
+ private long serviceId;
+
+ private List<OutOfBandManagementDriver> outOfBandManagementDrivers = new ArrayList<>();
+ private Map<String, OutOfBandManagementDriver> outOfBandManagementDriversMap = new HashMap<String, OutOfBandManagementDriver>();
+
+ private static final String OOBM_ENABLED_DETAIL = "outOfBandManagementEnabled";
+ private static final int ACQUIRE_GLOBAL_LOCK_TIMEOUT_FOR_HOST = 120;
+
+ private Cache<Long, Long> hostAlertCache;
+ private static ExecutorService backgroundSyncExecutor;
+
+ private String getOutOfBandManagementHostLock(long id) {
+ return "oobm.host." + id;
+ }
+
+ private void initializeDriversMap() {
+ if (outOfBandManagementDriversMap.isEmpty() && outOfBandManagementDrivers != null && outOfBandManagementDrivers.size() > 0) {
+ for (final OutOfBandManagementDriver driver : outOfBandManagementDrivers) {
+ outOfBandManagementDriversMap.put(driver.getName().toLowerCase(), driver);
+ }
+ LOG.debug("Discovered out-of-band management drivers configured in the OutOfBandManagementService");
+ }
+ }
+
+ private OutOfBandManagementDriver getDriver(final OutOfBandManagement outOfBandManagementConfig) {
+ if (!Strings.isNullOrEmpty(outOfBandManagementConfig.getDriver())) {
+ final OutOfBandManagementDriver driver = outOfBandManagementDriversMap.get(outOfBandManagementConfig.getDriver());
+ if (driver != null) {
+ return driver;
+ }
+ }
+ throw new CloudRuntimeException("Configured out-of-band management driver is not available. Aborting any out-of-band management action.");
+ }
+
+ protected OutOfBandManagement updateConfig(final OutOfBandManagement outOfBandManagementConfig, final ImmutableMap<OutOfBandManagement.Option, String> options) {
+ if (outOfBandManagementConfig == null) {
+ throw new CloudRuntimeException("Out-of-band management is not configured for the host. Aborting.");
+ }
+ if (options == null) {
+ return outOfBandManagementConfig;
+ }
+ for (OutOfBandManagement.Option option: options.keySet()) {
+ final String value = options.get(option);
+ if (Strings.isNullOrEmpty(value)) {
+ continue;
+ }
+ switch (option) {
+ case DRIVER:
+ outOfBandManagementConfig.setDriver(value);
+ break;
+ case ADDRESS:
+ outOfBandManagementConfig.setAddress(value);
+ break;
+ case PORT:
+ outOfBandManagementConfig.setPort(Integer.parseInt(value));
+ break;
+ case USERNAME:
+ outOfBandManagementConfig.setUsername(value);
+ break;
+ case PASSWORD:
+ outOfBandManagementConfig.setPassword(value);
+ break;
+ }
+ }
+ return outOfBandManagementConfig;
+ }
+
+ protected ImmutableMap<OutOfBandManagement.Option, String> getOptions(final OutOfBandManagement outOfBandManagementConfig) {
+ final ImmutableMap.Builder<OutOfBandManagement.Option, String> optionsBuilder = ImmutableMap.builder();
+ if (outOfBandManagementConfig == null) {
+ throw new CloudRuntimeException("Out-of-band management is not configured for the host. Aborting.");
+ }
+ for (OutOfBandManagement.Option option: OutOfBandManagement.Option.values()) {
+ String value = null;
+ switch (option) {
+ case DRIVER:
+ value = outOfBandManagementConfig.getDriver();
+ break;
+ case ADDRESS:
+ value = outOfBandManagementConfig.getAddress();
+ break;
+ case PORT:
+ if (outOfBandManagementConfig.getPort() != null) {
+ value = String.valueOf(outOfBandManagementConfig.getPort());
+ }
+ break;
+ case USERNAME:
+ value = outOfBandManagementConfig.getUsername();
+ break;
+ case PASSWORD:
+ value = outOfBandManagementConfig.getPassword();
+ break;
+ }
+ if (value != null) {
+ optionsBuilder.put(option, value);
+ }
+ }
+ return optionsBuilder.build();
+ }
+
+ private void sendAuthError(final Host host, final String message) {
+ try {
+ hostAlertCache.asMap().putIfAbsent(host.getId(), 0L);
+ Long sentCount = hostAlertCache.asMap().get(host.getId());
+ if (sentCount != null && sentCount <= 0) {
+ boolean concurrentUpdateResult = hostAlertCache.asMap().replace(host.getId(), sentCount, sentCount+1L);
+ if (concurrentUpdateResult) {
+ final String subject = String.format("Out-of-band management auth-error detected for host:%d in cluster:%d, zone:%d", host.getId(), host.getClusterId(), host.getDataCenterId());
+ LOG.error(subject + ": " + message);
+ alertMgr.sendAlert(AlertManager.AlertType.ALERT_TYPE_OOBM_AUTH_ERROR, host.getDataCenterId(), host.getPodId(), subject, message);
+ }
+ }
+ } catch (Exception ignored) {
+ }
+ }
+
+ private boolean transitionPowerState(OutOfBandManagement.PowerState.Event event, OutOfBandManagement outOfBandManagementHost) {
+ if (outOfBandManagementHost == null) {
+ return false;
+ }
+ OutOfBandManagement.PowerState currentPowerState = outOfBandManagementHost.getPowerState();
+ try {
+ OutOfBandManagement.PowerState newPowerState = OutOfBandManagement.PowerState.getStateMachine().getNextState(currentPowerState, event);
+ boolean result = outOfBandManagementDao.updateState(currentPowerState, event, newPowerState, outOfBandManagementHost, null);
+ if (result) {
+ final String message = String.format("Transitioned out-of-band management power state from:%s to:%s due to event:%s for the host id:%d", currentPowerState, newPowerState, event, outOfBandManagementHost.getHostId());
+ LOG.debug(message);
+ ActionEventUtils.onActionEvent(CallContext.current().getCallingUserId(), CallContext.current().getCallingAccountId(), Domain.ROOT_DOMAIN,
+ EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_POWERSTATE_TRANSITION, message);
+ }
+ return result;
+ } catch (NoTransitionException ignored) {
+ LOG.trace(String.format("Unable to transition out-of-band management power state for host id=%s for the event=%s and current power state=%s", outOfBandManagementHost.getHostId(), event, currentPowerState));
+ }
+ return false;
+ }
+
+ private boolean isOutOfBandManagementEnabledForZone(Long zoneId) {
+ if (zoneId == null) {
+ return true;
+ }
+ final DataCenterDetailVO zoneDetails = dataCenterDetailsDao.findDetail(zoneId, OOBM_ENABLED_DETAIL);
+ if (zoneDetails != null && !Strings.isNullOrEmpty(zoneDetails.getValue()) && !Boolean.valueOf(zoneDetails.getValue())) {
+ return false;
+ }
+ return true;
+ }
+
+ private boolean isOutOfBandManagementEnabledForCluster(Long clusterId) {
+ if (clusterId == null) {
+ return true;
+ }
+ final ClusterDetailsVO clusterDetails = clusterDetailsDao.findDetail(clusterId, OOBM_ENABLED_DETAIL);
+ if (clusterDetails != null && !Strings.isNullOrEmpty(clusterDetails.getValue()) && !Boolean.valueOf(clusterDetails.getValue())) {
+ return false;
+ }
+ return true;
+ }
+
+ private boolean isOutOfBandManagementEnabledForHost(Long hostId) {
+ if (hostId == null) {
+ return false;
+ }
+ final OutOfBandManagement outOfBandManagementConfig = outOfBandManagementDao.findByHost(hostId);
+ if (outOfBandManagementConfig == null || !outOfBandManagementConfig.isEnabled()) {
+ return false;
+ }
+ return true;
+ }
+
+ private void checkOutOfBandManagementEnabledByZoneClusterHost(final Host host) {
+ if (!isOutOfBandManagementEnabledForZone(host.getDataCenterId())) {
+ throw new CloudRuntimeException("Out-of-band management is disabled for the host's zone. Aborting Operation.");
+ }
+ if (!isOutOfBandManagementEnabledForCluster(host.getClusterId())) {
+ throw new CloudRuntimeException("Out-of-band management is disabled for the host's cluster. Aborting Operation.");
+ }
+ if (!isOutOfBandManagementEnabledForHost(host.getId())) {
+ throw new CloudRuntimeException("Out-of-band management is disabled or not configured for the host. Aborting Operation.");
+ }
+ }
+
+ public boolean isOutOfBandManagementEnabled(final Host host) {
+ return isOutOfBandManagementEnabledForZone(host.getDataCenterId())
+ && isOutOfBandManagementEnabledForCluster(host.getClusterId())
+ && isOutOfBandManagementEnabledForHost(host.getId());
+ }
+
+ public boolean transitionPowerStateToDisabled(List<? extends Host> hosts) {
+ boolean result = true;
+ for (Host host : hosts) {
+ result = result && transitionPowerState(OutOfBandManagement.PowerState.Event.Disabled,
+ outOfBandManagementDao.findByHost(host.getId()));
+ }
+ return result;
+ }
+
+ public void submitBackgroundPowerSyncTask(final Host host) {
+ if (host != null) {
+ // Note: This is a blocking queue based executor
+ backgroundSyncExecutor.submit(new OutOfBandManagementBackgroundTask(this, host, OutOfBandManagement.PowerOperation.STATUS));
+ }
+ }
+
+ private OutOfBandManagementResponse buildEnableDisableResponse(final boolean enabled) {
+ final OutOfBandManagementResponse response = new OutOfBandManagementResponse();
+ response.setEnabled(enabled);
+ response.setSuccess(true);
+ return response;
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_ENABLEDISABLE, eventDescription = "enabling/disabling out-of-band management on a zone")
+ public OutOfBandManagementResponse enableDisableOutOfBandManagement(final DataCenter zone, final boolean enabled) {
+ dataCenterDetailsDao.persist(zone.getId(), OOBM_ENABLED_DETAIL, String.valueOf(enabled));
+ if (!enabled) {
+ transitionPowerStateToDisabled(hostDao.findByDataCenterId(zone.getId()));
+ }
+ return buildEnableDisableResponse(enabled);
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_ENABLEDISABLE, eventDescription = "enabling/disabling out-of-band management on a cluster")
+ public OutOfBandManagementResponse enableDisableOutOfBandManagement(final Cluster cluster, final boolean enabled) {
+ clusterDetailsDao.persist(cluster.getId(), OOBM_ENABLED_DETAIL, String.valueOf(enabled));
+ if (!enabled) {
+ transitionPowerStateToDisabled(hostDao.findByClusterId(cluster.getId()));
+ }
+ return buildEnableDisableResponse(enabled);
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_ENABLEDISABLE, eventDescription = "enabling/disabling out-of-band management on a host")
+ public OutOfBandManagementResponse enableDisableOutOfBandManagement(final Host host, final boolean enabled) {
+ final OutOfBandManagement outOfBandManagementConfig = outOfBandManagementDao.findByHost(host.getId());
+ if (outOfBandManagementConfig == null) {
+ final OutOfBandManagementResponse response = new OutOfBandManagementResponse(null);
+ response.setSuccess(false);
+ response.setResultDescription("Out-of-band management is not configured for the host. Please configure the host before enabling/disabling it.");
+ return response;
+ }
+ hostAlertCache.invalidate(host.getId());
+ outOfBandManagementConfig.setEnabled(enabled);
+ boolean updateResult = outOfBandManagementDao.update(outOfBandManagementConfig.getId(), (OutOfBandManagementVO) outOfBandManagementConfig);
+ if (updateResult && !enabled) {
+ transitionPowerStateToDisabled(Collections.singletonList(host));
+ }
+ return buildEnableDisableResponse(enabled && updateResult);
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_CONFIGURE, eventDescription = "updating out-of-band management configuration")
+ public OutOfBandManagementResponse configureOutOfBandManagement(final Host host, final ImmutableMap<OutOfBandManagement.Option, String> options) {
+ OutOfBandManagement outOfBandManagementConfig = outOfBandManagementDao.findByHost(host.getId());
+ if (outOfBandManagementConfig == null) {
+ outOfBandManagementConfig = outOfBandManagementDao.persist(new OutOfBandManagementVO(host.getId()));
+ }
+ outOfBandManagementConfig = updateConfig(outOfBandManagementConfig, options);
+ if (Strings.isNullOrEmpty(outOfBandManagementConfig.getDriver()) || !outOfBandManagementDriversMap.containsKey(outOfBandManagementConfig.getDriver().toLowerCase())) {
+ throw new CloudRuntimeException("Out-of-band management driver is not available. Please provide a valid driver name.");
+ }
+
+ boolean updatedConfig = outOfBandManagementDao.update(outOfBandManagementConfig.getId(), (OutOfBandManagementVO) outOfBandManagementConfig);
+ CallContext.current().setEventDetails("host id:" + host.getId() + " configuration:" + outOfBandManagementConfig.getAddress() + ":" + outOfBandManagementConfig.getPort());
+
+ if (!updatedConfig) {
+ throw new CloudRuntimeException("Failed to update out-of-band management config for the host in the database.");
+ }
+
+ String result = "Out-of-band management successfully configured for the host";
+ LOG.debug(result);
+
+ final OutOfBandManagementResponse response = new OutOfBandManagementResponse(outOfBandManagementDao.findByHost(host.getId()));
+ response.setResultDescription(result);
+ response.setSuccess(true);
+ return response;
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_ACTION, eventDescription = "issuing Host out-of-band management action", async = true)
+ public OutOfBandManagementResponse executeOutOfBandManagementPowerOperation(final Host host, final OutOfBandManagement.PowerOperation powerOperation, final Long timeout) {
+ checkOutOfBandManagementEnabledByZoneClusterHost(host);
--- End diff --
Why do we throw a runtime exception when someone attempts to execute an OOBM operation against host that does not support it? Why not simple send a response that says OOBM is either disabled or not supported for the host? Of primary concern is that someone could DoS a management server by calling this API operation repeatedly on a disabled host and generating excess CPU and memory load for exception creation.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by DaanHoogland <gi...@git.apache.org>.
Github user DaanHoogland commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-213592386
@swill I am running the tests on this one. You want to both run it or beat me at it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61777665
--- Diff: api/src/org/apache/cloudstack/api/command/admin/outofbandmanagement/IssueOutOfBandManagementPowerActionCmd.java ---
@@ -0,0 +1,128 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.api.command.admin.outofbandmanagement;
+
+import com.cloud.event.EventTypes;
+import com.cloud.exception.ConcurrentOperationException;
+import com.cloud.exception.InsufficientCapacityException;
+import com.cloud.exception.NetworkRuleConflictException;
+import com.cloud.exception.ResourceAllocationException;
+import com.cloud.exception.ResourceUnavailableException;
+import com.cloud.host.Host;
+import com.google.common.base.Strings;
+import org.apache.cloudstack.acl.RoleType;
+import org.apache.cloudstack.api.APICommand;
+import org.apache.cloudstack.api.ApiCommandJobType;
+import org.apache.cloudstack.api.ApiConstants;
+import org.apache.cloudstack.api.ApiErrorCode;
+import org.apache.cloudstack.api.BaseAsyncCmd;
+import org.apache.cloudstack.api.Parameter;
+import org.apache.cloudstack.api.ServerApiException;
+import org.apache.cloudstack.api.response.HostResponse;
+import org.apache.cloudstack.api.response.OutOfBandManagementResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagement.PowerOperation;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagementService;
+
+import javax.inject.Inject;
+
+@APICommand(name = "issueOutOfBandManagementPowerAction", description = "Initiates the specified power action to the host's out-of-band management interface",
+ responseObject = OutOfBandManagementResponse.class, requestHasSensitiveInfo = false, responseHasSensitiveInfo = false, authorized = {RoleType.Admin})
+public class IssueOutOfBandManagementPowerActionCmd extends BaseAsyncCmd {
+ @Inject
+ private OutOfBandManagementService outOfBandManagementService;
+
+ /////////////////////////////////////////////////////
+ //////////////// API parameters /////////////////////
+ /////////////////////////////////////////////////////
+
+ @Parameter(name = ApiConstants.HOST_ID, type = CommandType.UUID, entityType = HostResponse.class, required = true, description = "the ID of the host")
+ private Long hostId;
+
+ @Parameter(name = ApiConstants.TIMEOUT, type = CommandType.LONG, description = "optional operation timeout in seconds that overrides the global or cluster-level out-of-band management timeout setting")
+ private Long actionTimeout;
+
+ @Parameter(name = ApiConstants.ACTION, type = CommandType.STRING, required = true, description = "out-of-band management power actions, valid actions are: ON, OFF, CYCLE, RESET, SOFT, STATUS")
+ private String outOfBandManagementPowerOperation;
+
+ /////////////////////////////////////////////////////
+ /////////////// API Implementation///////////////////
+ /////////////////////////////////////////////////////
+
+ @Override
+ public String getCommandName() {
+ return "issueoutofbandmanagementpoweractionresponse";
+ }
+
+ private void validateParams() {
+ if (getHostId() == null || getHostId() < 1L) {
+ throw new ServerApiException(ApiErrorCode.PARAM_ERROR, "Invalid host ID: " + getHostId());
+ }
+ if (Strings.isNullOrEmpty(getOutOfBandManagementPowerOperation())) {
+ throw new ServerApiException(ApiErrorCode.PARAM_ERROR, "Invalid out-of-band management power action: " + getOutOfBandManagementPowerOperation());
+ }
+ }
+
+ @Override
+ public void execute() throws ResourceUnavailableException, InsufficientCapacityException, ServerApiException, ConcurrentOperationException, ResourceAllocationException, NetworkRuleConflictException {
+ validateParams();
+ Host host = _resourceService.getHost(getHostId());
+ if (host == null) {
+ throw new ServerApiException(ApiErrorCode.PARAM_ERROR, "Unable to find host by ID: " + getHostId());
+ }
+ PowerOperation powerOperation = PowerOperation.valueOf(getOutOfBandManagementPowerOperation());
--- End diff --
Can this lookup be made case-insensitive?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61795629
--- Diff: server/src/org/apache/cloudstack/outofbandmanagement/OutOfBandManagementBackgroundTask.java ---
@@ -0,0 +1,45 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.outofbandmanagement;
+
+import com.cloud.host.Host;
+
+public class OutOfBandManagementBackgroundTask implements Runnable {
+ final private OutOfBandManagementService service;
+ final private Host host;
+ final private OutOfBandManagement.PowerOperation powerOperation;
+
+ public OutOfBandManagementBackgroundTask(OutOfBandManagementService service, Host host, OutOfBandManagement.PowerOperation powerOperation) {
+ this.service = service;
+ this.host = host;
+ this.powerOperation = powerOperation;
+ }
+
+ @Override
+ public String toString() {
+ return String.format("[OOBM Task] Power operation:%s on Host:%d(%s)", powerOperation, host.getId(), host.getName());
+ }
+
+ @Override
+ public void run() {
+ try {
+ service.executeOutOfBandManagementPowerOperation(host, powerOperation, null);
+ } catch (Exception ignored) {
--- End diff --
Log the ignored exception to ``WARN`` as it may be an indication of a emerging systematic failure.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-213617967
@DaanHoogland thanks, in your test environment you'll need to install ipmitool (yum/apt-get) and the oobm tests would pass. These tests are also run by Travis, and see they pass too; in travis's before_install.sh we install/setup ipmitool see the changes for details
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61813822
--- Diff: server/src/org/apache/cloudstack/outofbandmanagement/OutOfBandManagementServiceImpl.java ---
@@ -0,0 +1,532 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.outofbandmanagement;
+
+import com.cloud.alert.AlertManager;
+import com.cloud.dc.ClusterDetailsDao;
+import com.cloud.dc.ClusterDetailsVO;
+import com.cloud.dc.DataCenter;
+import com.cloud.dc.DataCenterDetailVO;
+import com.cloud.dc.dao.DataCenterDetailsDao;
+import com.cloud.domain.Domain;
+import com.cloud.event.ActionEvent;
+import com.cloud.event.ActionEventUtils;
+import com.cloud.event.EventTypes;
+import com.cloud.host.Host;
+import com.cloud.host.dao.HostDao;
+import com.cloud.org.Cluster;
+import com.cloud.utils.component.Manager;
+import com.cloud.utils.component.ManagerBase;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.db.Transaction;
+import com.cloud.utils.db.TransactionCallback;
+import com.cloud.utils.db.TransactionStatus;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.fsm.NoTransitionException;
+import com.google.common.base.Strings;
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import com.google.common.collect.ImmutableMap;
+import org.apache.cloudstack.api.response.OutOfBandManagementResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.framework.config.ConfigKey;
+import org.apache.cloudstack.framework.config.Configurable;
+import org.apache.cloudstack.outofbandmanagement.dao.OutOfBandManagementDao;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverChangePasswordCommand;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverPowerCommand;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverResponse;
+import org.apache.cloudstack.utils.identity.ManagementServerNode;
+import org.apache.log4j.Logger;
+import org.springframework.stereotype.Component;
+
+import javax.ejb.Local;
+import javax.inject.Inject;
+import javax.naming.ConfigurationException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+@Component
+@Local(value = {OutOfBandManagementService.class})
+public class OutOfBandManagementServiceImpl extends ManagerBase implements OutOfBandManagementService, Manager, Configurable {
+ public static final Logger LOG = Logger.getLogger(OutOfBandManagementServiceImpl.class);
+
+ @Inject
+ private ClusterDetailsDao clusterDetailsDao;
+ @Inject
+ private DataCenterDetailsDao dataCenterDetailsDao;
+ @Inject
+ private OutOfBandManagementDao outOfBandManagementDao;
+ @Inject
+ private HostDao hostDao;
+ @Inject
+ private AlertManager alertMgr;
+
+ private String name;
+ private long serviceId;
+
+ private List<OutOfBandManagementDriver> outOfBandManagementDrivers = new ArrayList<>();
+ private Map<String, OutOfBandManagementDriver> outOfBandManagementDriversMap = new HashMap<String, OutOfBandManagementDriver>();
+
+ private static final String OOBM_ENABLED_DETAIL = "outOfBandManagementEnabled";
+ private static final int ACQUIRE_GLOBAL_LOCK_TIMEOUT_FOR_HOST = 120;
+
+ private Cache<Long, Long> hostAlertCache;
+ private static ExecutorService backgroundSyncExecutor;
+
+ private String getOutOfBandManagementHostLock(long id) {
+ return "oobm.host." + id;
+ }
+
+ private void initializeDriversMap() {
+ if (outOfBandManagementDriversMap.isEmpty() && outOfBandManagementDrivers != null && outOfBandManagementDrivers.size() > 0) {
+ for (final OutOfBandManagementDriver driver : outOfBandManagementDrivers) {
+ outOfBandManagementDriversMap.put(driver.getName().toLowerCase(), driver);
+ }
+ LOG.debug("Discovered out-of-band management drivers configured in the OutOfBandManagementService");
+ }
+ }
+
+ private OutOfBandManagementDriver getDriver(final OutOfBandManagement outOfBandManagementConfig) {
+ if (!Strings.isNullOrEmpty(outOfBandManagementConfig.getDriver())) {
+ final OutOfBandManagementDriver driver = outOfBandManagementDriversMap.get(outOfBandManagementConfig.getDriver());
+ if (driver != null) {
+ return driver;
+ }
+ }
+ throw new CloudRuntimeException("Configured out-of-band management driver is not available. Aborting any out-of-band management action.");
+ }
+
+ protected OutOfBandManagement updateConfig(final OutOfBandManagement outOfBandManagementConfig, final ImmutableMap<OutOfBandManagement.Option, String> options) {
+ if (outOfBandManagementConfig == null) {
+ throw new CloudRuntimeException("Out-of-band management is not configured for the host. Aborting.");
+ }
+ if (options == null) {
+ return outOfBandManagementConfig;
+ }
+ for (OutOfBandManagement.Option option: options.keySet()) {
+ final String value = options.get(option);
+ if (Strings.isNullOrEmpty(value)) {
+ continue;
+ }
+ switch (option) {
+ case DRIVER:
+ outOfBandManagementConfig.setDriver(value);
+ break;
+ case ADDRESS:
+ outOfBandManagementConfig.setAddress(value);
+ break;
+ case PORT:
+ outOfBandManagementConfig.setPort(Integer.parseInt(value));
+ break;
+ case USERNAME:
+ outOfBandManagementConfig.setUsername(value);
+ break;
+ case PASSWORD:
+ outOfBandManagementConfig.setPassword(value);
+ break;
+ }
+ }
+ return outOfBandManagementConfig;
+ }
+
+ protected ImmutableMap<OutOfBandManagement.Option, String> getOptions(final OutOfBandManagement outOfBandManagementConfig) {
+ final ImmutableMap.Builder<OutOfBandManagement.Option, String> optionsBuilder = ImmutableMap.builder();
+ if (outOfBandManagementConfig == null) {
+ throw new CloudRuntimeException("Out-of-band management is not configured for the host. Aborting.");
+ }
+ for (OutOfBandManagement.Option option: OutOfBandManagement.Option.values()) {
+ String value = null;
+ switch (option) {
+ case DRIVER:
+ value = outOfBandManagementConfig.getDriver();
+ break;
+ case ADDRESS:
+ value = outOfBandManagementConfig.getAddress();
+ break;
+ case PORT:
+ if (outOfBandManagementConfig.getPort() != null) {
+ value = String.valueOf(outOfBandManagementConfig.getPort());
+ }
+ break;
+ case USERNAME:
+ value = outOfBandManagementConfig.getUsername();
+ break;
+ case PASSWORD:
+ value = outOfBandManagementConfig.getPassword();
+ break;
+ }
+ if (value != null) {
+ optionsBuilder.put(option, value);
+ }
+ }
+ return optionsBuilder.build();
+ }
+
+ private void sendAuthError(final Host host, final String message) {
+ try {
+ hostAlertCache.asMap().putIfAbsent(host.getId(), 0L);
+ Long sentCount = hostAlertCache.asMap().get(host.getId());
+ if (sentCount != null && sentCount <= 0) {
+ boolean concurrentUpdateResult = hostAlertCache.asMap().replace(host.getId(), sentCount, sentCount+1L);
+ if (concurrentUpdateResult) {
+ final String subject = String.format("Out-of-band management auth-error detected for host:%d in cluster:%d, zone:%d", host.getId(), host.getClusterId(), host.getDataCenterId());
+ LOG.error(subject + ": " + message);
+ alertMgr.sendAlert(AlertManager.AlertType.ALERT_TYPE_OOBM_AUTH_ERROR, host.getDataCenterId(), host.getPodId(), subject, message);
+ }
+ }
+ } catch (Exception ignored) {
+ }
+ }
+
+ private boolean transitionPowerState(OutOfBandManagement.PowerState.Event event, OutOfBandManagement outOfBandManagementHost) {
+ if (outOfBandManagementHost == null) {
+ return false;
+ }
+ OutOfBandManagement.PowerState currentPowerState = outOfBandManagementHost.getPowerState();
+ try {
+ OutOfBandManagement.PowerState newPowerState = OutOfBandManagement.PowerState.getStateMachine().getNextState(currentPowerState, event);
+ boolean result = outOfBandManagementDao.updateState(currentPowerState, event, newPowerState, outOfBandManagementHost, null);
+ if (result) {
+ final String message = String.format("Transitioned out-of-band management power state from:%s to:%s due to event:%s for the host id:%d", currentPowerState, newPowerState, event, outOfBandManagementHost.getHostId());
+ LOG.debug(message);
+ ActionEventUtils.onActionEvent(CallContext.current().getCallingUserId(), CallContext.current().getCallingAccountId(), Domain.ROOT_DOMAIN,
+ EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_POWERSTATE_TRANSITION, message);
+ }
+ return result;
+ } catch (NoTransitionException ignored) {
+ LOG.trace(String.format("Unable to transition out-of-band management power state for host id=%s for the event=%s and current power state=%s", outOfBandManagementHost.getHostId(), event, currentPowerState));
+ }
+ return false;
+ }
+
+ private boolean isOutOfBandManagementEnabledForZone(Long zoneId) {
+ if (zoneId == null) {
+ return true;
+ }
+ final DataCenterDetailVO zoneDetails = dataCenterDetailsDao.findDetail(zoneId, OOBM_ENABLED_DETAIL);
+ if (zoneDetails != null && !Strings.isNullOrEmpty(zoneDetails.getValue()) && !Boolean.valueOf(zoneDetails.getValue())) {
+ return false;
+ }
+ return true;
+ }
+
+ private boolean isOutOfBandManagementEnabledForCluster(Long clusterId) {
+ if (clusterId == null) {
+ return true;
+ }
+ final ClusterDetailsVO clusterDetails = clusterDetailsDao.findDetail(clusterId, OOBM_ENABLED_DETAIL);
+ if (clusterDetails != null && !Strings.isNullOrEmpty(clusterDetails.getValue()) && !Boolean.valueOf(clusterDetails.getValue())) {
+ return false;
+ }
+ return true;
+ }
+
+ private boolean isOutOfBandManagementEnabledForHost(Long hostId) {
+ if (hostId == null) {
+ return false;
+ }
+ final OutOfBandManagement outOfBandManagementConfig = outOfBandManagementDao.findByHost(hostId);
+ if (outOfBandManagementConfig == null || !outOfBandManagementConfig.isEnabled()) {
+ return false;
+ }
+ return true;
+ }
+
+ private void checkOutOfBandManagementEnabledByZoneClusterHost(final Host host) {
+ if (!isOutOfBandManagementEnabledForZone(host.getDataCenterId())) {
+ throw new CloudRuntimeException("Out-of-band management is disabled for the host's zone. Aborting Operation.");
+ }
+ if (!isOutOfBandManagementEnabledForCluster(host.getClusterId())) {
+ throw new CloudRuntimeException("Out-of-band management is disabled for the host's cluster. Aborting Operation.");
+ }
+ if (!isOutOfBandManagementEnabledForHost(host.getId())) {
+ throw new CloudRuntimeException("Out-of-band management is disabled or not configured for the host. Aborting Operation.");
+ }
+ }
+
+ public boolean isOutOfBandManagementEnabled(final Host host) {
+ return isOutOfBandManagementEnabledForZone(host.getDataCenterId())
+ && isOutOfBandManagementEnabledForCluster(host.getClusterId())
+ && isOutOfBandManagementEnabledForHost(host.getId());
+ }
+
+ public boolean transitionPowerStateToDisabled(List<? extends Host> hosts) {
+ boolean result = true;
+ for (Host host : hosts) {
+ result = result && transitionPowerState(OutOfBandManagement.PowerState.Event.Disabled,
+ outOfBandManagementDao.findByHost(host.getId()));
+ }
+ return result;
+ }
+
+ public void submitBackgroundPowerSyncTask(final Host host) {
+ if (host != null) {
+ // Note: This is a blocking queue based executor
+ backgroundSyncExecutor.submit(new OutOfBandManagementBackgroundTask(this, host, OutOfBandManagement.PowerOperation.STATUS));
+ }
+ }
+
+ private OutOfBandManagementResponse buildEnableDisableResponse(final boolean enabled) {
+ final OutOfBandManagementResponse response = new OutOfBandManagementResponse();
+ response.setEnabled(enabled);
+ response.setSuccess(true);
+ return response;
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_ENABLEDISABLE, eventDescription = "enabling/disabling out-of-band management on a zone")
+ public OutOfBandManagementResponse enableDisableOutOfBandManagement(final DataCenter zone, final boolean enabled) {
+ dataCenterDetailsDao.persist(zone.getId(), OOBM_ENABLED_DETAIL, String.valueOf(enabled));
+ if (!enabled) {
+ transitionPowerStateToDisabled(hostDao.findByDataCenterId(zone.getId()));
+ }
+ return buildEnableDisableResponse(enabled);
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_ENABLEDISABLE, eventDescription = "enabling/disabling out-of-band management on a cluster")
+ public OutOfBandManagementResponse enableDisableOutOfBandManagement(final Cluster cluster, final boolean enabled) {
+ clusterDetailsDao.persist(cluster.getId(), OOBM_ENABLED_DETAIL, String.valueOf(enabled));
+ if (!enabled) {
+ transitionPowerStateToDisabled(hostDao.findByClusterId(cluster.getId()));
+ }
+ return buildEnableDisableResponse(enabled);
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_ENABLEDISABLE, eventDescription = "enabling/disabling out-of-band management on a host")
+ public OutOfBandManagementResponse enableDisableOutOfBandManagement(final Host host, final boolean enabled) {
+ final OutOfBandManagement outOfBandManagementConfig = outOfBandManagementDao.findByHost(host.getId());
+ if (outOfBandManagementConfig == null) {
+ final OutOfBandManagementResponse response = new OutOfBandManagementResponse(null);
+ response.setSuccess(false);
+ response.setResultDescription("Out-of-band management is not configured for the host. Please configure the host before enabling/disabling it.");
+ return response;
+ }
+ hostAlertCache.invalidate(host.getId());
+ outOfBandManagementConfig.setEnabled(enabled);
+ boolean updateResult = outOfBandManagementDao.update(outOfBandManagementConfig.getId(), (OutOfBandManagementVO) outOfBandManagementConfig);
+ if (updateResult && !enabled) {
+ transitionPowerStateToDisabled(Collections.singletonList(host));
+ }
+ return buildEnableDisableResponse(enabled && updateResult);
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_CONFIGURE, eventDescription = "updating out-of-band management configuration")
+ public OutOfBandManagementResponse configureOutOfBandManagement(final Host host, final ImmutableMap<OutOfBandManagement.Option, String> options) {
+ OutOfBandManagement outOfBandManagementConfig = outOfBandManagementDao.findByHost(host.getId());
+ if (outOfBandManagementConfig == null) {
+ outOfBandManagementConfig = outOfBandManagementDao.persist(new OutOfBandManagementVO(host.getId()));
+ }
+ outOfBandManagementConfig = updateConfig(outOfBandManagementConfig, options);
+ if (Strings.isNullOrEmpty(outOfBandManagementConfig.getDriver()) || !outOfBandManagementDriversMap.containsKey(outOfBandManagementConfig.getDriver().toLowerCase())) {
+ throw new CloudRuntimeException("Out-of-band management driver is not available. Please provide a valid driver name.");
+ }
+
+ boolean updatedConfig = outOfBandManagementDao.update(outOfBandManagementConfig.getId(), (OutOfBandManagementVO) outOfBandManagementConfig);
+ CallContext.current().setEventDetails("host id:" + host.getId() + " configuration:" + outOfBandManagementConfig.getAddress() + ":" + outOfBandManagementConfig.getPort());
+
+ if (!updatedConfig) {
+ throw new CloudRuntimeException("Failed to update out-of-band management config for the host in the database.");
+ }
+
+ String result = "Out-of-band management successfully configured for the host";
+ LOG.debug(result);
+
+ final OutOfBandManagementResponse response = new OutOfBandManagementResponse(outOfBandManagementDao.findByHost(host.getId()));
+ response.setResultDescription(result);
+ response.setSuccess(true);
+ return response;
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_ACTION, eventDescription = "issuing Host out-of-band management action", async = true)
+ public OutOfBandManagementResponse executeOutOfBandManagementPowerOperation(final Host host, final OutOfBandManagement.PowerOperation powerOperation, final Long timeout) {
+ checkOutOfBandManagementEnabledByZoneClusterHost(host);
+ final OutOfBandManagement outOfBandManagementConfig = outOfBandManagementDao.findByHost(host.getId());
+ final ImmutableMap<OutOfBandManagement.Option, String> options = getOptions(outOfBandManagementConfig);
+ final OutOfBandManagementDriver driver = getDriver(outOfBandManagementConfig);
+
+ Long actionTimeOut = timeout;
+ if (actionTimeOut == null) {
+ actionTimeOut = OutOfBandManagementActionTimeout.valueIn(host.getClusterId());
+ }
+
+ final OutOfBandManagementDriverPowerCommand cmd = new OutOfBandManagementDriverPowerCommand(options, actionTimeOut, powerOperation);
+ final OutOfBandManagementDriverResponse driverResponse = driver.execute(cmd);
+
+ if (driverResponse == null) {
+ throw new CloudRuntimeException(String.format("Out-of-band Management action (%s) on host (%s) failed due to no response from the driver", powerOperation, host.getUuid()));
+ }
+
+ if (powerOperation.equals(OutOfBandManagement.PowerOperation.STATUS)) {
+ transitionPowerState(driverResponse.toEvent(), outOfBandManagementConfig);
+ }
+
+ if (!driverResponse.isSuccess()) {
+ String errorMessage = String.format("Out-of-band Management action (%s) on host (%s) failed with error: %s", powerOperation, host.getUuid(), driverResponse.getError());
+ if (driverResponse.hasAuthFailure()) {
+ errorMessage = String.format("Out-of-band Management action (%s) on host (%s) failed due to authentication error: %s. Please check configured credentials.", powerOperation, host.getUuid(), driverResponse.getError());
+ sendAuthError(host, errorMessage);
+ }
+ if (!powerOperation.equals(OutOfBandManagement.PowerOperation.STATUS)) {
+ LOG.debug(errorMessage);
+ }
+ throw new CloudRuntimeException(errorMessage);
+ }
+
+ final OutOfBandManagementResponse response = new OutOfBandManagementResponse(outOfBandManagementDao.findByHost(host.getId()));
+ response.setSuccess(driverResponse.isSuccess());
+ response.setResultDescription(driverResponse.getResult());
+ response.setId(host.getUuid());
+ response.setOutOfBandManagementAction(powerOperation.toString());
+ return response;
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_CHANGE_PASSWORD, eventDescription = "updating out-of-band management password")
+ public OutOfBandManagementResponse changeOutOfBandManagementPassword(final Host host, final String newPassword) {
+ checkOutOfBandManagementEnabledByZoneClusterHost(host);
+ if (Strings.isNullOrEmpty(newPassword)) {
+ throw new CloudRuntimeException(String.format("Cannot change out-of-band management password as provided new-password is null or empty for the host %s.", host.getUuid()));
+ }
+ GlobalLock outOfBandManagementHostLock = GlobalLock.getInternLock(getOutOfBandManagementHostLock(host.getId()));
+ try {
+ if (outOfBandManagementHostLock.lock(ACQUIRE_GLOBAL_LOCK_TIMEOUT_FOR_HOST)) {
+ try {
+ final OutOfBandManagement outOfBandManagementConfig = outOfBandManagementDao.findByHost(host.getId());
+
+ final ImmutableMap<OutOfBandManagement.Option, String> options = getOptions(outOfBandManagementConfig);
+ if (!(options.containsKey(OutOfBandManagement.Option.PASSWORD) && !Strings.isNullOrEmpty(options.get(OutOfBandManagement.Option.PASSWORD)))) {
+ throw new CloudRuntimeException(String.format("Cannot change out-of-band management password as we've no previously configured password for the host %s.", host.getUuid()));
+ }
+ final OutOfBandManagementDriver driver = getDriver(outOfBandManagementConfig);
+
+ final OutOfBandManagementDriverChangePasswordCommand cmd = new OutOfBandManagementDriverChangePasswordCommand(options, OutOfBandManagementActionTimeout.valueIn(host.getClusterId()), newPassword);
+ final OutOfBandManagementDriverResponse driverResponse;
+ try {
+ driverResponse = driver.execute(cmd);
+ } catch (Exception e) {
+ LOG.error("Out-of-band management change password failed due to driver error: " + e.getMessage());
+ throw new CloudRuntimeException(String.format("Failed to change out-of-band management password for host (%s) due to driver error: %s", host.getUuid(), e.getMessage()));
+ }
+
+ if (!driverResponse.isSuccess()) {
+ throw new CloudRuntimeException(String.format("Failed to change out-of-band management password for host (%s) with error: %s", host.getUuid(), driverResponse.getError()));
+ }
+
+ final boolean updatedConfigResult = Transaction.execute(new TransactionCallback<Boolean>() {
+ @Override
+ public Boolean doInTransaction(TransactionStatus status) {
+ OutOfBandManagement updatedOutOfBandManagementConfig = outOfBandManagementDao.findByHost(host.getId());
+ updatedOutOfBandManagementConfig.setPassword(newPassword);
+ return outOfBandManagementDao.update(updatedOutOfBandManagementConfig.getId(), (OutOfBandManagementVO) updatedOutOfBandManagementConfig);
+ }
+ });
+
+ if (!updatedConfigResult) {
+ LOG.error(String.format("Succeeded to change out-of-band management password but failed to updated in database the new password:%s for the host id:%d", newPassword, host.getId()));
+ }
+
+ final OutOfBandManagementResponse response = new OutOfBandManagementResponse();
+ response.setSuccess(updatedConfigResult && driverResponse.isSuccess());
+ response.setResultDescription(driverResponse.getResult());
+ response.setId(host.getUuid());
+ return response;
+ } finally {
+ outOfBandManagementHostLock.unlock();
+ }
+ } else {
+ LOG.error("Unable to acquire synchronization lock to change out-of-band management password for host id: " + host.getId());
+ throw new CloudRuntimeException(String.format("Unable to acquire lock to change out-of-band management password for host (%s), please try after some time.", host.getUuid()));
+ }
+ } finally {
+ outOfBandManagementHostLock.releaseRef();
+ }
+ }
+
+ @Override
+ public String getName() {
+ return name;
+ }
+
+ @Override
+ public long getId() {
+ return serviceId;
+ }
+
+ @Override
+ public boolean configure(final String name, final Map<String, Object> params) throws ConfigurationException {
+ this.name = name;
+ this.serviceId = ManagementServerNode.getManagementServerId();
+
+ final int poolSize = OutOfBandManagementSyncThreadPoolSize.value();
+
+ hostAlertCache = CacheBuilder.newBuilder()
+ .concurrencyLevel(4)
+ .weakKeys()
+ .maximumSize(100 * poolSize)
+ .expireAfterWrite(1, TimeUnit.DAYS)
+ .build();
+
+ backgroundSyncExecutor = new ThreadPoolExecutor(poolSize, poolSize,
+ 0L, TimeUnit.MILLISECONDS,
+ new ArrayBlockingQueue<Runnable>(10 * poolSize, true), new ThreadPoolExecutor.CallerRunsPolicy());
+
+ LOG.info("Starting out-of-band management background sync executor with thread pool-size=" + poolSize + " and background sync thread interval=" + OutOfBandManagementSyncThreadInterval.value() + "s");
+ return true;
+ }
+
+ @Override
+ public boolean start() {
+ initializeDriversMap();
+ return true;
+ }
+
+ @Override
+ public boolean stop() {
+ backgroundSyncExecutor.shutdown();
+ outOfBandManagementDao.expireOutOfBandManagementOwnershipByServer(getId());
+ return true;
+ }
+
+ @Override
+ public String getConfigComponentName() {
+ return OutOfBandManagementServiceImpl.class.getSimpleName();
+ }
+
+ @Override
+ public ConfigKey<?>[] getConfigKeys() {
+ return new ConfigKey<?>[] {OutOfBandManagementActionTimeout, OutOfBandManagementSyncThreadInterval, OutOfBandManagementSyncThreadPoolSize};
+ }
+
+ public List<OutOfBandManagementDriver> getOutOfBandManagementDrivers() {
+ return outOfBandManagementDrivers;
+ }
+
+ public void setOutOfBandManagementDrivers(List<OutOfBandManagementDriver> outOfBandManagementDrivers) {
+ this.outOfBandManagementDrivers = outOfBandManagementDrivers;
--- End diff --
Make a defensive copy of ``outOfBandManagementDrivers`` to ensure changes made by the caller do not impact this object.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61792886
--- Diff: plugins/outofbandmanagement-drivers/ipmitool/test/org/apache/cloudstack/outofbandmanagement/driver/ipmitool/IpmitoolWrapperTest.java ---
@@ -0,0 +1,107 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+
+package org.apache.cloudstack.outofbandmanagement.driver.ipmitool;
+
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.google.common.collect.ImmutableMap;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagement;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverResponse;
+import org.junit.Assert;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.mockito.runners.MockitoJUnitRunner;
+
+import java.util.Arrays;
+import java.util.List;
+
+@RunWith(MockitoJUnitRunner.class)
+public class IpmitoolWrapperTest {
+ @Test
+ public void testParsePowerCommandValid() {
+ Assert.assertEquals(IpmitoolWrapper.parsePowerCommand(OutOfBandManagement.PowerOperation.ON), "on");
+ Assert.assertEquals(IpmitoolWrapper.parsePowerCommand(OutOfBandManagement.PowerOperation.OFF), "off");
+ Assert.assertEquals(IpmitoolWrapper.parsePowerCommand(OutOfBandManagement.PowerOperation.CYCLE), "cycle");
+ Assert.assertEquals(IpmitoolWrapper.parsePowerCommand(OutOfBandManagement.PowerOperation.RESET), "reset");
+ Assert.assertEquals(IpmitoolWrapper.parsePowerCommand(OutOfBandManagement.PowerOperation.SOFT), "soft");
+ Assert.assertEquals(IpmitoolWrapper.parsePowerCommand(OutOfBandManagement.PowerOperation.STATUS), "status");
+ }
+
+ @Test
+ public void testParsePowerCommandInvalid() {
+ try {
+ IpmitoolWrapper.parsePowerCommand(null);
+ Assert.fail("IpmitoolWrapper.parsePowerCommand failed to throw exception on invalid power state");
+ } catch (IllegalStateException e) {
+ }
+ }
+
+ @Test
+ public void testParsePowerState() {
+ Assert.assertEquals(IpmitoolWrapper.parsePowerState(null), OutOfBandManagement.PowerState.Unknown);
+ Assert.assertEquals(IpmitoolWrapper.parsePowerState(""), OutOfBandManagement.PowerState.Unknown);
+ Assert.assertEquals(IpmitoolWrapper.parsePowerState(" "), OutOfBandManagement.PowerState.Unknown);
+ Assert.assertEquals(IpmitoolWrapper.parsePowerState("invalid data"), OutOfBandManagement.PowerState.Unknown);
+ Assert.assertEquals(IpmitoolWrapper.parsePowerState("Chassis Power is on"), OutOfBandManagement.PowerState.On);
+ Assert.assertEquals(IpmitoolWrapper.parsePowerState("Chassis Power is off"), OutOfBandManagement.PowerState.Off);
+ }
+
+ @Test
+ public void testGetIpmiToolCommandArgs() {
+ List<String> args = IpmitoolWrapper.getIpmiToolCommandArgs("binpath", "intf", "1", null);
+ assert args != null;
+ Assert.assertEquals(args.size(), 6);
+ Assert.assertArrayEquals(args.toArray(), new String[]{"binpath", "-I", "intf", "-R", "1", "-v"});
+
+ ImmutableMap.Builder<OutOfBandManagement.Option, String> argMap = new ImmutableMap.Builder<>();
+ argMap.put(OutOfBandManagement.Option.DRIVER, "ipmitool");
+ argMap.put(OutOfBandManagement.Option.ADDRESS, "127.0.0.1");
+ List<String> argsWithOpts = IpmitoolWrapper.getIpmiToolCommandArgs("binpath", "intf", "1", argMap.build(), "user", "list");
+ assert argsWithOpts != null;
+ Assert.assertEquals(argsWithOpts.size(), 10);
+ Assert.assertArrayEquals(argsWithOpts.toArray(), new String[]{"binpath", "-I", "intf", "-R", "1", "-v", "-H", "127.0.0.1", "user", "list"});
+ }
+
+ @Test
+ public void testFindIpmiUser() {
+ // Invalid data
+ try {
+ IpmitoolWrapper.findIpmiUser("some invalid string\n", "admin");
+ Assert.fail("IpmitoolWrapper.findIpmiUser failed to throw exception on invalid data");
+ } catch (CloudRuntimeException ignore) {
+ }
--- End diff --
Consider splitting the success and failure test cases into separate methods with the failure method using the ``@Test(expected=CloudRuntimeException.class)`` annotation.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61776712
--- Diff: api/src/org/apache/cloudstack/api/command/admin/outofbandmanagement/ConfigureOutOfBandManagementCmd.java ---
@@ -0,0 +1,127 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.api.command.admin.outofbandmanagement;
+
+import com.cloud.exception.ConcurrentOperationException;
+import com.cloud.exception.InsufficientCapacityException;
+import com.cloud.exception.NetworkRuleConflictException;
+import com.cloud.exception.ResourceAllocationException;
+import com.cloud.exception.ResourceUnavailableException;
+import com.cloud.host.Host;
+import com.google.common.base.Strings;
+import com.google.common.collect.ImmutableMap;
+import org.apache.cloudstack.acl.RoleType;
+import org.apache.cloudstack.api.APICommand;
+import org.apache.cloudstack.api.ApiConstants;
+import org.apache.cloudstack.api.ApiErrorCode;
+import org.apache.cloudstack.api.BaseCmd;
+import org.apache.cloudstack.api.Parameter;
+import org.apache.cloudstack.api.ServerApiException;
+import org.apache.cloudstack.api.response.HostResponse;
+import org.apache.cloudstack.api.response.OutOfBandManagementResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagement;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagementService;
+
+import javax.inject.Inject;
+
+@APICommand(name = "configureOutOfBandManagement", description = "Configures a host's out-of-band management interface",
--- End diff --
Please add the version to this annotation
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r60885367
--- Diff: api/src/org/apache/cloudstack/api/command/admin/outofbandmanagement/ConfigureOutOfBandManagementCmd.java ---
@@ -0,0 +1,127 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.api.command.admin.outofbandmanagement;
+
+import com.cloud.exception.ConcurrentOperationException;
+import com.cloud.exception.InsufficientCapacityException;
+import com.cloud.exception.NetworkRuleConflictException;
+import com.cloud.exception.ResourceAllocationException;
+import com.cloud.exception.ResourceUnavailableException;
+import com.cloud.host.Host;
+import com.google.common.base.Strings;
+import com.google.common.collect.ImmutableMap;
+import org.apache.cloudstack.acl.RoleType;
+import org.apache.cloudstack.api.APICommand;
+import org.apache.cloudstack.api.ApiConstants;
+import org.apache.cloudstack.api.ApiErrorCode;
+import org.apache.cloudstack.api.BaseCmd;
+import org.apache.cloudstack.api.Parameter;
+import org.apache.cloudstack.api.ServerApiException;
+import org.apache.cloudstack.api.response.HostResponse;
+import org.apache.cloudstack.api.response.OutOfBandManagementResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagement;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagementService;
+
+import javax.inject.Inject;
+
+@APICommand(name = "configureOutOfBandManagement", description = "Configures a host's out-of-band management interface",
+ responseObject = OutOfBandManagementResponse.class, requestHasSensitiveInfo = true, responseHasSensitiveInfo = false, authorized = {RoleType.Admin})
+public class ConfigureOutOfBandManagementCmd extends BaseCmd {
+ @Inject
+ private OutOfBandManagementService outOfBandManagementService;
+
+ /////////////////////////////////////////////////////
+ //////////////// API parameters /////////////////////
+ /////////////////////////////////////////////////////
+
+ @Parameter(name = ApiConstants.HOST_ID, type = CommandType.UUID, entityType = HostResponse.class, required = true, description = "the ID of the host")
+ private Long hostId;
+
+ @Parameter(name = ApiConstants.DRIVER, type = CommandType.STRING, required = true, description = "the host management interface driver, for example: ipmitool")
+ private String driver;
+
+ @Parameter(name = ApiConstants.IP_ADDRESS, type = CommandType.STRING, required = true, description = "the host management interface IP address")
+ private String address;
+
+ @Parameter(name = ApiConstants.PORT, type = CommandType.STRING, required = true, description = "the host management interface port")
--- End diff --
Thanks, will fix this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-216907666
@swill the issue is actually with ipmitool binary itself https://bugzilla.redhat.com/show_bug.cgi?id=1286035 where it may sometimes fail due to errors in ipmitool's handling of sessions. I'll see if I can work around that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
GitHub user rhtyd reopened a pull request:
https://github.com/apache/cloudstack/pull/1502
CLOUDSTACK-9299: Out-of-band Management for CloudStack
Support access to a host\u2019s out-of-band management interface (e.g. IPMI, iLO,
DRAC, etc.) to manage host power operations (on/off etc.) and querying current
power state in CloudStack.
Given the wide range of out-of-band management interfaces such as iLO and iDRA,
the service implementation allows for development of separate drivers as plugins.
This feature comes with a ipmitool based driver that uses the
ipmitool (http://linux.die.net/man/1/ipmitool) to communicate with any
out-of-band management interface that support IPMI 2.0.
This feature allows following common use-cases:
- Restarting stalled/failed hosts
- Powering off under-utilised hosts
- Powering on hosts for provisioning or to increase capacity
- Allowing system administrators to see the current power state of the host
For testing this feature, please install `ipmitool` (using yum/apt/brew) and `ipmisim`:
https://pypi.python.org/pypi/ipmisim
The default ipmitool location is assumed in /usr/bin, if this is different in your env please fix the global setting, see FS for details on various global settings.
FS:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Out-of-band+Management+for+CloudStack
/cc @jburwell @swill @abhinandanprateek @murali-reddy @borisstoyanov
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/shapeblue/cloudstack outofband-master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/cloudstack/pull/1502.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1502
----
commit 07564469e9a2ddd0334e8bdd30deed3ed53c3f09
Author: Rohit Yadav <ro...@shapeblue.com>
Date: 2015-12-28T11:07:03Z
CLOUDSTACK-9299: Out-of-band Management for CloudStack
Support access to a host\u2019s out-of-band management interface (e.g. IPMI, iLO,
DRAC, etc.) to manage host power operations (on/off etc.) and querying current
power state in CloudStack.
Given the wide range of out-of-band management interfaces such as iLO and iDRA,
the service implementation allows for development of separate drivers as plugins.
This feature comes with a ipmitool based driver that uses the
ipmitool (http://linux.die.net/man/1/ipmitool) to communicate with any
out-of-band management interface that support IPMI 2.0.
This feature allows following common use-cases:
- Restarting stalled/failed hosts
- Powering off under-utilised hosts
- Powering on hosts for provisioning or to increase capacity
- Allowing system administrators to see the current power state of the host
For testing this feature `ipmisim` can be used:
https://pypi.python.org/pypi/ipmisim
FS:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Out-of-band+Management+for+CloudStack
Signed-off-by: Rohit Yadav <ro...@shapeblue.com>
commit 4d5e8df2f952c4e0594c8fc4d11181f6d3da3811
Author: Rohit Yadav <ro...@shapeblue.com>
Date: 2016-05-06T00:22:07Z
travis: Use patched version of ipmitool for tests
- For out-of-band management feature (CLOUDSTACK-9299) use patched version of
ipmitool that would work on trusty travis machines
- The ipmitool used is from xenial/16.04 release with patch from RedHat
https://bugzilla.redhat.com/show_bug.cgi?id=1286035
- Installs ipmitool from xenial repositories to get all the dependencies
and then install patched deb version
- Skip test if the known failure occurs
Signed-off-by: Rohit Yadav <ro...@shapeblue.com>
commit e122bbfbfcf13dcec6e4848e8a8b99990047d060
Author: Rohit Yadav <ro...@shapeblue.com>
Date: 2016-05-11T08:59:22Z
HypervisorUtilsTest: increate timeout to 8seconds
Increases timeout to a larger value to avoid failures in VM environments such as
TravisCI.
Signed-off-by: Rohit Yadav <ro...@shapeblue.com>
commit 6135f6d98fa6b363b0468c17a8e713aeb5054437
Author: Rohit Yadav <ro...@shapeblue.com>
Date: 2016-05-12T05:34:37Z
CLOUDSTACK-9378: Fix for #1497
Reorder cleanup items so cleanup won't fail
Signed-off-by: Rohit Yadav <ro...@shapeblue.com>
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r62321576
--- Diff: api/src/org/apache/cloudstack/outofbandmanagement/OutOfBandManagementService.java ---
@@ -0,0 +1,51 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.outofbandmanagement;
+
+import com.cloud.dc.DataCenter;
+import com.cloud.host.Host;
+import com.cloud.org.Cluster;
+import com.google.common.collect.ImmutableMap;
+import org.apache.cloudstack.api.response.OutOfBandManagementResponse;
+import org.apache.cloudstack.framework.config.ConfigKey;
+
+import java.util.List;
+
+public interface OutOfBandManagementService {
+
+ ConfigKey<Long> OutOfBandManagementActionTimeout = new ConfigKey<Long>("Advanced", Long.class, "outofbandmanagement.action.timeout", "60",
+ "The out of band management action timeout in seconds, configurable by cluster", true, ConfigKey.Scope.Cluster);
+
+ ConfigKey<Long> OutOfBandManagementSyncThreadInterval = new ConfigKey<Long>("Advanced", Long.class, "outofbandmanagement.sync.interval", "300000",
+ "The interval (in milliseconds) when the out-of-band management background sync are retrieved", true, ConfigKey.Scope.Global);
+
+ ConfigKey<Integer> OutOfBandManagementSyncThreadPoolSize = new ConfigKey<Integer>("Advanced", Integer.class, "outofbandmanagement.sync.poolsize", "50",
+ "The out of band management background sync thread pool size", true, ConfigKey.Scope.Global);
+
+ long getId();
+ boolean isOutOfBandManagementEnabled(Host host);
+ void submitBackgroundPowerSyncTask(Host host);
+ boolean transitionPowerStateToDisabled(List<? extends Host> hosts);
--- End diff --
@rhtyd the issue is that it is a wider type specification that is necessary. It can also cause erasure compiler errors. Therefore, we should only be using <? extends> when it is truly necessary. In this case, the ``OutOfBandManagementService`` only operates on instances of ``Host``.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r62442087
--- Diff: utils/src/main/java/org/apache/cloudstack/utils/process/ProcessRunner.java ---
@@ -0,0 +1,111 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+
+package org.apache.cloudstack.utils.process;
+
+import org.apache.log4j.Logger;
+
+import java.io.IOException;
+import java.util.List;
+
+public class ProcessRunner {
+ public static final Logger LOG = Logger.getLogger(ProcessRunner.class);
+
+ private String stdOutput;
+ private String stdError;
+ private int returnCode = -1;
+
+ public String getStdOutput() {
+ return stdOutput;
+ }
+
+ public void setStdOutput(String stdOutput) {
+ this.stdOutput = stdOutput;
+ }
+
+ public String getStdError() {
+ return stdError;
+ }
+
+ public void setStdError(String stdError) {
+ this.stdError = stdError;
+ }
+
+ public int getReturnCode() {
+ return returnCode;
+ }
+
+ public void setReturnCode(int returnCode) {
+ this.returnCode = returnCode;
+ }
+
+ public static ProcessRunner executeCommands(List<String> commands, long timeOutSeconds) {
--- End diff --
Fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-218784278
@swill @rhtyd LGTM based on code review
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by swill <gi...@git.apache.org>.
Github user swill commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-217271201
So is this going to be another reason Travis fails randomly? :( If we know it is going to fail randomly, we probably want to remove it from the Travis tests to make sure it does not create false negatives for everyone. If it fails the tests in travis, does that mean that the functionality also randomly fails?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r62790898
--- Diff: utils/src/main/java/org/apache/cloudstack/utils/process/ProcessRunner.java ---
@@ -0,0 +1,112 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+
+package org.apache.cloudstack.utils.process;
+
+import com.cloud.utils.concurrency.NamedThreadFactory;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Strings;
+import org.apache.log4j.Logger;
+import org.joda.time.Duration;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.List;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+public class ProcessRunner {
+ public static final Logger LOG = Logger.getLogger(ProcessRunner.class);
+
+ private static final ExecutorService processExecutor = Executors.newCachedThreadPool(new NamedThreadFactory("ProcessRunner"));
--- End diff --
@jburwell I'll modify the executeCommands method to accept an executor; the ipmitool driver would create and re-use a fixedThreadPool
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r62321858
--- Diff: server/src/com/cloud/server/StatsCollector.java ---
@@ -251,8 +262,9 @@ public boolean start() {
}
private void init(Map<String, String> configs) {
- _executor = Executors.newScheduledThreadPool(4, new NamedThreadFactory("StatsCollector"));
+ _executor = Executors.newScheduledThreadPool(6, new NamedThreadFactory("StatsCollector"));
--- End diff --
I understand that ``6`` is in the ``init`` method. My question is how did we arrive at that number? Why is 6 instead of 2 or 10?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61846300
--- Diff: server/src/org/apache/cloudstack/outofbandmanagement/OutOfBandManagementBackgroundTask.java ---
@@ -0,0 +1,45 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.outofbandmanagement;
+
+import com.cloud.host.Host;
+
+public class OutOfBandManagementBackgroundTask implements Runnable {
+ final private OutOfBandManagementService service;
+ final private Host host;
+ final private OutOfBandManagement.PowerOperation powerOperation;
+
+ public OutOfBandManagementBackgroundTask(OutOfBandManagementService service, Host host, OutOfBandManagement.PowerOperation powerOperation) {
+ this.service = service;
+ this.host = host;
+ this.powerOperation = powerOperation;
+ }
+
+ @Override
+ public String toString() {
+ return String.format("[OOBM Task] Power operation:%s on Host:%d(%s)", powerOperation, host.getId(), host.getName());
+ }
+
+ @Override
+ public void run() {
+ try {
+ service.executeOutOfBandManagementPowerOperation(host, powerOperation, null);
+ } catch (Exception ignored) {
--- End diff --
fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by borisstoyanov <gi...@git.apache.org>.
Github user borisstoyanov commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-216271049
Marvin tests where successful. Testing community submission of OOBM, which includes end to end regression testing with IPMI simulator. Issue power state commands and stage transition, configuration and password management and testing of feature enablement on separate layers of infrastructure.
LGTM.
```
==== Marvin Init Successful ====
=== TestName: test_oobm_background_powerstate_sync | Status : SUCCESS ===
=== TestName: test_oobm_change_password | Status : SUCCESS ===
=== TestName: test_oobm_configure_default_driver | Status : SUCCESS ===
=== TestName: test_oobm_configure_invalid_driver | Status : SUCCESS ===
=== TestName: test_oobm_disable_feature_invalid | Status : SUCCESS ===
=== TestName: test_oobm_disable_feature_valid | Status : SUCCESS ===
=== TestName: test_oobm_enable_feature_invalid | Status : SUCCESS ===
=== TestName: test_oobm_enable_feature_valid | Status : SUCCESS ===
=== TestName: test_oobm_enabledisable_across_clusterzones | Status : SUCCESS ===
=== TestName: test_oobm_issue_power_cycle | Status : SUCCESS ===
=== TestName: test_oobm_issue_power_off | Status : SUCCESS ===
=== TestName: test_oobm_issue_power_on | Status : SUCCESS ===
=== TestName: test_oobm_issue_power_reset | Status : SUCCESS ===
=== TestName: test_oobm_issue_power_soft | Status : SUCCESS ===
=== TestName: test_oobm_issue_power_status | Status : SUCCESS ===
=== TestName: test_oobm_multiple_mgmt_server_ownership | Status : SUCCESS ===
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61776838
--- Diff: api/src/org/apache/cloudstack/api/command/admin/outofbandmanagement/DisableOutOfBandManagementForClusterCmd.java ---
@@ -0,0 +1,105 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.api.command.admin.outofbandmanagement;
+
+import com.cloud.event.EventTypes;
+import com.cloud.exception.ConcurrentOperationException;
+import com.cloud.exception.InsufficientCapacityException;
+import com.cloud.exception.NetworkRuleConflictException;
+import com.cloud.exception.ResourceAllocationException;
+import com.cloud.exception.ResourceUnavailableException;
+import com.cloud.org.Cluster;
+import org.apache.cloudstack.acl.RoleType;
+import org.apache.cloudstack.api.APICommand;
+import org.apache.cloudstack.api.ApiConstants;
+import org.apache.cloudstack.api.ApiErrorCode;
+import org.apache.cloudstack.api.BaseAsyncCmd;
+import org.apache.cloudstack.api.BaseCmd;
+import org.apache.cloudstack.api.Parameter;
+import org.apache.cloudstack.api.ServerApiException;
+import org.apache.cloudstack.api.response.ClusterResponse;
+import org.apache.cloudstack.api.response.OutOfBandManagementResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagementService;
+
+import javax.inject.Inject;
+
+@APICommand(name = "disableOutOfBandManagementForCluster", description = "Disables out-of-band management for a cluster",
--- End diff --
Please add the version to this annotation
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61776673
--- Diff: api/src/org/apache/cloudstack/api/command/admin/outofbandmanagement/ChangeOutOfBandManagementPasswordCmd.java ---
@@ -0,0 +1,116 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.api.command.admin.outofbandmanagement;
+
+import com.cloud.event.EventTypes;
+import com.cloud.exception.ConcurrentOperationException;
+import com.cloud.exception.InsufficientCapacityException;
+import com.cloud.exception.NetworkRuleConflictException;
+import com.cloud.exception.ResourceAllocationException;
+import com.cloud.exception.ResourceUnavailableException;
+import com.cloud.host.Host;
+import com.google.common.base.Strings;
+import org.apache.cloudstack.acl.RoleType;
+import org.apache.cloudstack.api.APICommand;
+import org.apache.cloudstack.api.ApiConstants;
+import org.apache.cloudstack.api.ApiErrorCode;
+import org.apache.cloudstack.api.BaseAsyncCmd;
+import org.apache.cloudstack.api.Parameter;
+import org.apache.cloudstack.api.ServerApiException;
+import org.apache.cloudstack.api.response.HostResponse;
+import org.apache.cloudstack.api.response.OutOfBandManagementResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagementService;
+
+import javax.inject.Inject;
+
+@APICommand(name = "changeOutOfBandManagementPassword", description = "Changes out-of-band management interface password on the host and updates the interface configuration in CloudStack if the operation succeeds, else reverts the old password",
--- End diff --
Please add the version to this annotation
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r62661909
--- Diff: utils/src/main/java/org/apache/cloudstack/utils/process/ProcessRunner.java ---
@@ -0,0 +1,112 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+
+package org.apache.cloudstack.utils.process;
+
+import com.cloud.utils.concurrency.NamedThreadFactory;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Strings;
+import org.apache.log4j.Logger;
+import org.joda.time.Duration;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.List;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+public class ProcessRunner {
+ public static final Logger LOG = Logger.getLogger(ProcessRunner.class);
+
+ private static final ExecutorService processExecutor = Executors.newCachedThreadPool(new NamedThreadFactory("ProcessRunner"));
--- End diff --
Reuse does not prevent overwhelming the management server. A flood of new requests would spawn an explosion of thread creation. A ``fixedThreadPool`` or a bounded blocking queue would protect against such a scenario.
One approach would be to be convert make this class instanceable and pass in a pool size via a constructor. The OOBM subsystem could then initialize it with a global setting or a sane default value.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61845431
--- Diff: engine/schema/src/org/apache/cloudstack/outofbandmanagement/dao/OutOfBandManagementDaoImpl.java ---
@@ -0,0 +1,163 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.outofbandmanagement.dao;
+
+import com.cloud.utils.DateUtil;
+import com.cloud.utils.db.Attribute;
+import com.cloud.utils.db.DB;
+import com.cloud.utils.db.Filter;
+import com.cloud.utils.db.GenericDaoBase;
+import com.cloud.utils.db.SearchBuilder;
+import com.cloud.utils.db.SearchCriteria;
+import com.cloud.utils.db.TransactionLegacy;
+import com.cloud.utils.db.UpdateBuilder;
+import com.cloud.utils.exception.CloudRuntimeException;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagement;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagementVO;
+import org.apache.log4j.Logger;
+import org.springframework.stereotype.Component;
+
+import javax.ejb.Local;
+import java.sql.PreparedStatement;
+import java.sql.SQLException;
+import java.util.List;
+
+@DB
+@Component
+@Local(value = {OutOfBandManagementDao.class})
+public class OutOfBandManagementDaoImpl extends GenericDaoBase<OutOfBandManagementVO, Long> implements OutOfBandManagementDao {
+ private static final Logger LOG = Logger.getLogger(OutOfBandManagementDaoImpl.class);
+
+ private SearchBuilder<OutOfBandManagementVO> HostSearch;
+ private SearchBuilder<OutOfBandManagementVO> ManagementServerSearch;
+ private SearchBuilder<OutOfBandManagementVO> OutOfBandManagementOwnerSearch;
+ private SearchBuilder<OutOfBandManagementVO> StateUpdateSearch;
+
+ private Attribute PowerStateAttr;
+ private Attribute MsIdAttr;
+ private Attribute UpdateTimeAttr;
+
+ public OutOfBandManagementDaoImpl() {
+ super();
+
+ HostSearch = createSearchBuilder();
+ HostSearch.and("hostId", HostSearch.entity().getHostId(), SearchCriteria.Op.EQ);
+ HostSearch.done();
+
+ ManagementServerSearch = createSearchBuilder();
+ ManagementServerSearch.and("server", ManagementServerSearch.entity().getManagementServerId(), SearchCriteria.Op.EQ);
+ ManagementServerSearch.done();
+
+ OutOfBandManagementOwnerSearch = createSearchBuilder();
+ OutOfBandManagementOwnerSearch.and("server", OutOfBandManagementOwnerSearch.entity().getManagementServerId(), SearchCriteria.Op.EQ);
+ OutOfBandManagementOwnerSearch.or("serverNull", OutOfBandManagementOwnerSearch.entity().getManagementServerId(), SearchCriteria.Op.NULL);
+ OutOfBandManagementOwnerSearch.done();
+
+ StateUpdateSearch = createSearchBuilder();
+ StateUpdateSearch.and("status", StateUpdateSearch.entity().getPowerState(), SearchCriteria.Op.EQ);
+ StateUpdateSearch.and("id", StateUpdateSearch.entity().getId(), SearchCriteria.Op.EQ);
+ StateUpdateSearch.and("update", StateUpdateSearch.entity().getUpdateCount(), SearchCriteria.Op.EQ);
+ StateUpdateSearch.done();
+
+ PowerStateAttr = _allAttributes.get("powerState");
+ MsIdAttr = _allAttributes.get("managementServerId");
+ UpdateTimeAttr = _allAttributes.get("updateTime");
+ assert (PowerStateAttr != null && MsIdAttr != null && UpdateTimeAttr != null) : "Couldn't find one of these attributes";
+ }
+
+ @Override
+ public OutOfBandManagement findByHost(long hostId) {
+ SearchCriteria<OutOfBandManagementVO> sc = HostSearch.create("hostId", hostId);
+ return findOneBy(sc);
+ }
+
+ @Override
+ public List<OutOfBandManagementVO> findAllByManagementServer(long serverId) {
+ SearchCriteria<OutOfBandManagementVO> sc = OutOfBandManagementOwnerSearch.create();
+ sc.setParameters("server", serverId);
+ return listBy(sc, new Filter(OutOfBandManagementVO.class, "updateTime", true, null, null));
+ }
+
+ private void executeExpireOwnershipSql(final String sql, long resource) {
+ TransactionLegacy txn = TransactionLegacy.currentTxn();
+ try {
+ txn.start();
+ PreparedStatement pstmt = txn.prepareAutoCloseStatement(sql);
+ pstmt.setLong(1, resource);
+ pstmt.executeUpdate();
+ txn.commit();
+ } catch (SQLException e) {
+ txn.rollback();
+ throw new CloudRuntimeException("Unable to reset out-of-band management ownership based on resource:" + resource);
+ }
+ }
+
+ @Override
+ public void expireOutOfBandManagementOwnershipByHours(long hours) {
+ final String resetOwnerSql = "UPDATE oobm set mgmt_server_id=NULL where update_time<= (NOW() - INTERVAL ? HOUR)";
+ executeExpireOwnershipSql(resetOwnerSql, hours);
+ }
+
+ @Override
+ public void expireOutOfBandManagementOwnershipByServer(long serverId) {
+ final String resetOwnerSql = "UPDATE oobm set mgmt_server_id=NULL, power_state=NULL where mgmt_server_id=?";
+ executeExpireOwnershipSql(resetOwnerSql, serverId);
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Expired out-of-band management ownership for hosts owned by management server id:" + serverId);
+ }
+ }
+
+ @Override
+ public boolean updateState(OutOfBandManagement.PowerState oldStatus, OutOfBandManagement.PowerState.Event event, OutOfBandManagement.PowerState newStatus, OutOfBandManagement vo, Object data) {
+ // lock target row from beginning to avoid lock-promotion caused deadlock
+ OutOfBandManagementVO oobmHost = lockRow(vo.getId(), true);
+ if (oobmHost == null) {
+ if (LOG.isTraceEnabled()) {
+ LOG.trace("Failed to lock row to update state for out-of-band management host id: " + vo.getHostId());
+ }
+ return false;
+ }
+
+ Long newManagementServerId = event.getServerId();
+ // Avoid updates when old ownership and state are same as new
+ if (oldStatus == newStatus && (oobmHost.getManagementServerId() != null && oobmHost.getManagementServerId().equals(newManagementServerId))) {
+ return false;
+ }
+
+ if (event == OutOfBandManagement.PowerState.Event.Disabled) {
+ newManagementServerId = null;
+ }
+
+ SearchCriteria<OutOfBandManagementVO> sc = StateUpdateSearch.create();
+ sc.setParameters("status", oldStatus);
+ sc.setParameters("id", oobmHost.getId());
+ sc.setParameters("update", oobmHost.getUpdateCount());
+
+ oobmHost.incrUpdateCount();
+ UpdateBuilder ub = getUpdateBuilder(oobmHost);
+ ub.set(oobmHost, PowerStateAttr, newStatus);
+ ub.set(oobmHost, UpdateTimeAttr, DateUtil.currentGMTTime());
+ ub.set(oobmHost, MsIdAttr, newManagementServerId);
--- End diff --
Read previous comment, let me know if you have more questions.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61846451
--- Diff: server/src/org/apache/cloudstack/outofbandmanagement/OutOfBandManagementServiceImpl.java ---
@@ -0,0 +1,532 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.outofbandmanagement;
+
+import com.cloud.alert.AlertManager;
+import com.cloud.dc.ClusterDetailsDao;
+import com.cloud.dc.ClusterDetailsVO;
+import com.cloud.dc.DataCenter;
+import com.cloud.dc.DataCenterDetailVO;
+import com.cloud.dc.dao.DataCenterDetailsDao;
+import com.cloud.domain.Domain;
+import com.cloud.event.ActionEvent;
+import com.cloud.event.ActionEventUtils;
+import com.cloud.event.EventTypes;
+import com.cloud.host.Host;
+import com.cloud.host.dao.HostDao;
+import com.cloud.org.Cluster;
+import com.cloud.utils.component.Manager;
+import com.cloud.utils.component.ManagerBase;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.db.Transaction;
+import com.cloud.utils.db.TransactionCallback;
+import com.cloud.utils.db.TransactionStatus;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.fsm.NoTransitionException;
+import com.google.common.base.Strings;
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import com.google.common.collect.ImmutableMap;
+import org.apache.cloudstack.api.response.OutOfBandManagementResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.framework.config.ConfigKey;
+import org.apache.cloudstack.framework.config.Configurable;
+import org.apache.cloudstack.outofbandmanagement.dao.OutOfBandManagementDao;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverChangePasswordCommand;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverPowerCommand;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverResponse;
+import org.apache.cloudstack.utils.identity.ManagementServerNode;
+import org.apache.log4j.Logger;
+import org.springframework.stereotype.Component;
+
+import javax.ejb.Local;
+import javax.inject.Inject;
+import javax.naming.ConfigurationException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+@Component
+@Local(value = {OutOfBandManagementService.class})
+public class OutOfBandManagementServiceImpl extends ManagerBase implements OutOfBandManagementService, Manager, Configurable {
+ public static final Logger LOG = Logger.getLogger(OutOfBandManagementServiceImpl.class);
+
+ @Inject
+ private ClusterDetailsDao clusterDetailsDao;
+ @Inject
+ private DataCenterDetailsDao dataCenterDetailsDao;
+ @Inject
+ private OutOfBandManagementDao outOfBandManagementDao;
+ @Inject
+ private HostDao hostDao;
+ @Inject
+ private AlertManager alertMgr;
+
+ private String name;
+ private long serviceId;
+
+ private List<OutOfBandManagementDriver> outOfBandManagementDrivers = new ArrayList<>();
+ private Map<String, OutOfBandManagementDriver> outOfBandManagementDriversMap = new HashMap<String, OutOfBandManagementDriver>();
+
+ private static final String OOBM_ENABLED_DETAIL = "outOfBandManagementEnabled";
+ private static final int ACQUIRE_GLOBAL_LOCK_TIMEOUT_FOR_HOST = 120;
+
+ private Cache<Long, Long> hostAlertCache;
+ private static ExecutorService backgroundSyncExecutor;
+
+ private String getOutOfBandManagementHostLock(long id) {
+ return "oobm.host." + id;
+ }
+
+ private void initializeDriversMap() {
+ if (outOfBandManagementDriversMap.isEmpty() && outOfBandManagementDrivers != null && outOfBandManagementDrivers.size() > 0) {
+ for (final OutOfBandManagementDriver driver : outOfBandManagementDrivers) {
+ outOfBandManagementDriversMap.put(driver.getName().toLowerCase(), driver);
+ }
+ LOG.debug("Discovered out-of-band management drivers configured in the OutOfBandManagementService");
+ }
+ }
+
+ private OutOfBandManagementDriver getDriver(final OutOfBandManagement outOfBandManagementConfig) {
+ if (!Strings.isNullOrEmpty(outOfBandManagementConfig.getDriver())) {
+ final OutOfBandManagementDriver driver = outOfBandManagementDriversMap.get(outOfBandManagementConfig.getDriver());
+ if (driver != null) {
+ return driver;
+ }
+ }
+ throw new CloudRuntimeException("Configured out-of-band management driver is not available. Aborting any out-of-band management action.");
+ }
+
+ protected OutOfBandManagement updateConfig(final OutOfBandManagement outOfBandManagementConfig, final ImmutableMap<OutOfBandManagement.Option, String> options) {
+ if (outOfBandManagementConfig == null) {
+ throw new CloudRuntimeException("Out-of-band management is not configured for the host. Aborting.");
+ }
+ if (options == null) {
+ return outOfBandManagementConfig;
+ }
+ for (OutOfBandManagement.Option option: options.keySet()) {
+ final String value = options.get(option);
+ if (Strings.isNullOrEmpty(value)) {
+ continue;
+ }
+ switch (option) {
+ case DRIVER:
+ outOfBandManagementConfig.setDriver(value);
+ break;
+ case ADDRESS:
+ outOfBandManagementConfig.setAddress(value);
+ break;
+ case PORT:
+ outOfBandManagementConfig.setPort(Integer.parseInt(value));
+ break;
+ case USERNAME:
+ outOfBandManagementConfig.setUsername(value);
+ break;
+ case PASSWORD:
+ outOfBandManagementConfig.setPassword(value);
+ break;
+ }
+ }
+ return outOfBandManagementConfig;
+ }
+
+ protected ImmutableMap<OutOfBandManagement.Option, String> getOptions(final OutOfBandManagement outOfBandManagementConfig) {
+ final ImmutableMap.Builder<OutOfBandManagement.Option, String> optionsBuilder = ImmutableMap.builder();
+ if (outOfBandManagementConfig == null) {
+ throw new CloudRuntimeException("Out-of-band management is not configured for the host. Aborting.");
+ }
+ for (OutOfBandManagement.Option option: OutOfBandManagement.Option.values()) {
+ String value = null;
+ switch (option) {
+ case DRIVER:
+ value = outOfBandManagementConfig.getDriver();
+ break;
+ case ADDRESS:
+ value = outOfBandManagementConfig.getAddress();
+ break;
+ case PORT:
+ if (outOfBandManagementConfig.getPort() != null) {
+ value = String.valueOf(outOfBandManagementConfig.getPort());
+ }
+ break;
+ case USERNAME:
+ value = outOfBandManagementConfig.getUsername();
+ break;
+ case PASSWORD:
+ value = outOfBandManagementConfig.getPassword();
+ break;
+ }
+ if (value != null) {
+ optionsBuilder.put(option, value);
+ }
+ }
+ return optionsBuilder.build();
+ }
+
+ private void sendAuthError(final Host host, final String message) {
+ try {
+ hostAlertCache.asMap().putIfAbsent(host.getId(), 0L);
+ Long sentCount = hostAlertCache.asMap().get(host.getId());
+ if (sentCount != null && sentCount <= 0) {
+ boolean concurrentUpdateResult = hostAlertCache.asMap().replace(host.getId(), sentCount, sentCount+1L);
+ if (concurrentUpdateResult) {
+ final String subject = String.format("Out-of-band management auth-error detected for host:%d in cluster:%d, zone:%d", host.getId(), host.getClusterId(), host.getDataCenterId());
+ LOG.error(subject + ": " + message);
+ alertMgr.sendAlert(AlertManager.AlertType.ALERT_TYPE_OOBM_AUTH_ERROR, host.getDataCenterId(), host.getPodId(), subject, message);
+ }
+ }
+ } catch (Exception ignored) {
+ }
+ }
+
+ private boolean transitionPowerState(OutOfBandManagement.PowerState.Event event, OutOfBandManagement outOfBandManagementHost) {
+ if (outOfBandManagementHost == null) {
+ return false;
+ }
+ OutOfBandManagement.PowerState currentPowerState = outOfBandManagementHost.getPowerState();
+ try {
+ OutOfBandManagement.PowerState newPowerState = OutOfBandManagement.PowerState.getStateMachine().getNextState(currentPowerState, event);
+ boolean result = outOfBandManagementDao.updateState(currentPowerState, event, newPowerState, outOfBandManagementHost, null);
+ if (result) {
+ final String message = String.format("Transitioned out-of-band management power state from:%s to:%s due to event:%s for the host id:%d", currentPowerState, newPowerState, event, outOfBandManagementHost.getHostId());
+ LOG.debug(message);
+ ActionEventUtils.onActionEvent(CallContext.current().getCallingUserId(), CallContext.current().getCallingAccountId(), Domain.ROOT_DOMAIN,
+ EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_POWERSTATE_TRANSITION, message);
+ }
+ return result;
+ } catch (NoTransitionException ignored) {
+ LOG.trace(String.format("Unable to transition out-of-band management power state for host id=%s for the event=%s and current power state=%s", outOfBandManagementHost.getHostId(), event, currentPowerState));
--- End diff --
in case a state does not change, we 'll get these errors a lot -- therefore a trace is used.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61847434
--- Diff: server/src/org/apache/cloudstack/outofbandmanagement/OutOfBandManagementServiceImpl.java ---
@@ -0,0 +1,532 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.outofbandmanagement;
+
+import com.cloud.alert.AlertManager;
+import com.cloud.dc.ClusterDetailsDao;
+import com.cloud.dc.ClusterDetailsVO;
+import com.cloud.dc.DataCenter;
+import com.cloud.dc.DataCenterDetailVO;
+import com.cloud.dc.dao.DataCenterDetailsDao;
+import com.cloud.domain.Domain;
+import com.cloud.event.ActionEvent;
+import com.cloud.event.ActionEventUtils;
+import com.cloud.event.EventTypes;
+import com.cloud.host.Host;
+import com.cloud.host.dao.HostDao;
+import com.cloud.org.Cluster;
+import com.cloud.utils.component.Manager;
+import com.cloud.utils.component.ManagerBase;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.db.Transaction;
+import com.cloud.utils.db.TransactionCallback;
+import com.cloud.utils.db.TransactionStatus;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.fsm.NoTransitionException;
+import com.google.common.base.Strings;
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import com.google.common.collect.ImmutableMap;
+import org.apache.cloudstack.api.response.OutOfBandManagementResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.framework.config.ConfigKey;
+import org.apache.cloudstack.framework.config.Configurable;
+import org.apache.cloudstack.outofbandmanagement.dao.OutOfBandManagementDao;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverChangePasswordCommand;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverPowerCommand;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverResponse;
+import org.apache.cloudstack.utils.identity.ManagementServerNode;
+import org.apache.log4j.Logger;
+import org.springframework.stereotype.Component;
+
+import javax.ejb.Local;
+import javax.inject.Inject;
+import javax.naming.ConfigurationException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+@Component
+@Local(value = {OutOfBandManagementService.class})
+public class OutOfBandManagementServiceImpl extends ManagerBase implements OutOfBandManagementService, Manager, Configurable {
+ public static final Logger LOG = Logger.getLogger(OutOfBandManagementServiceImpl.class);
+
+ @Inject
+ private ClusterDetailsDao clusterDetailsDao;
+ @Inject
+ private DataCenterDetailsDao dataCenterDetailsDao;
+ @Inject
+ private OutOfBandManagementDao outOfBandManagementDao;
+ @Inject
+ private HostDao hostDao;
+ @Inject
+ private AlertManager alertMgr;
+
+ private String name;
+ private long serviceId;
+
+ private List<OutOfBandManagementDriver> outOfBandManagementDrivers = new ArrayList<>();
+ private Map<String, OutOfBandManagementDriver> outOfBandManagementDriversMap = new HashMap<String, OutOfBandManagementDriver>();
+
+ private static final String OOBM_ENABLED_DETAIL = "outOfBandManagementEnabled";
+ private static final int ACQUIRE_GLOBAL_LOCK_TIMEOUT_FOR_HOST = 120;
+
+ private Cache<Long, Long> hostAlertCache;
+ private static ExecutorService backgroundSyncExecutor;
+
+ private String getOutOfBandManagementHostLock(long id) {
+ return "oobm.host." + id;
+ }
+
+ private void initializeDriversMap() {
+ if (outOfBandManagementDriversMap.isEmpty() && outOfBandManagementDrivers != null && outOfBandManagementDrivers.size() > 0) {
+ for (final OutOfBandManagementDriver driver : outOfBandManagementDrivers) {
+ outOfBandManagementDriversMap.put(driver.getName().toLowerCase(), driver);
+ }
+ LOG.debug("Discovered out-of-band management drivers configured in the OutOfBandManagementService");
+ }
+ }
+
+ private OutOfBandManagementDriver getDriver(final OutOfBandManagement outOfBandManagementConfig) {
+ if (!Strings.isNullOrEmpty(outOfBandManagementConfig.getDriver())) {
+ final OutOfBandManagementDriver driver = outOfBandManagementDriversMap.get(outOfBandManagementConfig.getDriver());
+ if (driver != null) {
+ return driver;
+ }
+ }
+ throw new CloudRuntimeException("Configured out-of-band management driver is not available. Aborting any out-of-band management action.");
+ }
+
+ protected OutOfBandManagement updateConfig(final OutOfBandManagement outOfBandManagementConfig, final ImmutableMap<OutOfBandManagement.Option, String> options) {
+ if (outOfBandManagementConfig == null) {
+ throw new CloudRuntimeException("Out-of-band management is not configured for the host. Aborting.");
+ }
+ if (options == null) {
+ return outOfBandManagementConfig;
+ }
+ for (OutOfBandManagement.Option option: options.keySet()) {
+ final String value = options.get(option);
+ if (Strings.isNullOrEmpty(value)) {
+ continue;
+ }
+ switch (option) {
+ case DRIVER:
+ outOfBandManagementConfig.setDriver(value);
+ break;
+ case ADDRESS:
+ outOfBandManagementConfig.setAddress(value);
+ break;
+ case PORT:
+ outOfBandManagementConfig.setPort(Integer.parseInt(value));
+ break;
+ case USERNAME:
+ outOfBandManagementConfig.setUsername(value);
+ break;
+ case PASSWORD:
+ outOfBandManagementConfig.setPassword(value);
+ break;
+ }
+ }
+ return outOfBandManagementConfig;
+ }
+
+ protected ImmutableMap<OutOfBandManagement.Option, String> getOptions(final OutOfBandManagement outOfBandManagementConfig) {
+ final ImmutableMap.Builder<OutOfBandManagement.Option, String> optionsBuilder = ImmutableMap.builder();
+ if (outOfBandManagementConfig == null) {
+ throw new CloudRuntimeException("Out-of-band management is not configured for the host. Aborting.");
+ }
+ for (OutOfBandManagement.Option option: OutOfBandManagement.Option.values()) {
+ String value = null;
+ switch (option) {
+ case DRIVER:
+ value = outOfBandManagementConfig.getDriver();
+ break;
+ case ADDRESS:
+ value = outOfBandManagementConfig.getAddress();
+ break;
+ case PORT:
+ if (outOfBandManagementConfig.getPort() != null) {
+ value = String.valueOf(outOfBandManagementConfig.getPort());
+ }
+ break;
+ case USERNAME:
+ value = outOfBandManagementConfig.getUsername();
+ break;
+ case PASSWORD:
+ value = outOfBandManagementConfig.getPassword();
+ break;
+ }
+ if (value != null) {
+ optionsBuilder.put(option, value);
+ }
+ }
+ return optionsBuilder.build();
+ }
+
+ private void sendAuthError(final Host host, final String message) {
+ try {
+ hostAlertCache.asMap().putIfAbsent(host.getId(), 0L);
+ Long sentCount = hostAlertCache.asMap().get(host.getId());
+ if (sentCount != null && sentCount <= 0) {
+ boolean concurrentUpdateResult = hostAlertCache.asMap().replace(host.getId(), sentCount, sentCount+1L);
+ if (concurrentUpdateResult) {
+ final String subject = String.format("Out-of-band management auth-error detected for host:%d in cluster:%d, zone:%d", host.getId(), host.getClusterId(), host.getDataCenterId());
+ LOG.error(subject + ": " + message);
+ alertMgr.sendAlert(AlertManager.AlertType.ALERT_TYPE_OOBM_AUTH_ERROR, host.getDataCenterId(), host.getPodId(), subject, message);
+ }
+ }
+ } catch (Exception ignored) {
+ }
+ }
+
+ private boolean transitionPowerState(OutOfBandManagement.PowerState.Event event, OutOfBandManagement outOfBandManagementHost) {
+ if (outOfBandManagementHost == null) {
+ return false;
+ }
+ OutOfBandManagement.PowerState currentPowerState = outOfBandManagementHost.getPowerState();
+ try {
+ OutOfBandManagement.PowerState newPowerState = OutOfBandManagement.PowerState.getStateMachine().getNextState(currentPowerState, event);
+ boolean result = outOfBandManagementDao.updateState(currentPowerState, event, newPowerState, outOfBandManagementHost, null);
+ if (result) {
+ final String message = String.format("Transitioned out-of-band management power state from:%s to:%s due to event:%s for the host id:%d", currentPowerState, newPowerState, event, outOfBandManagementHost.getHostId());
+ LOG.debug(message);
+ ActionEventUtils.onActionEvent(CallContext.current().getCallingUserId(), CallContext.current().getCallingAccountId(), Domain.ROOT_DOMAIN,
+ EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_POWERSTATE_TRANSITION, message);
+ }
+ return result;
+ } catch (NoTransitionException ignored) {
+ LOG.trace(String.format("Unable to transition out-of-band management power state for host id=%s for the event=%s and current power state=%s", outOfBandManagementHost.getHostId(), event, currentPowerState));
+ }
+ return false;
+ }
+
+ private boolean isOutOfBandManagementEnabledForZone(Long zoneId) {
+ if (zoneId == null) {
+ return true;
+ }
+ final DataCenterDetailVO zoneDetails = dataCenterDetailsDao.findDetail(zoneId, OOBM_ENABLED_DETAIL);
+ if (zoneDetails != null && !Strings.isNullOrEmpty(zoneDetails.getValue()) && !Boolean.valueOf(zoneDetails.getValue())) {
+ return false;
+ }
+ return true;
+ }
+
+ private boolean isOutOfBandManagementEnabledForCluster(Long clusterId) {
+ if (clusterId == null) {
+ return true;
+ }
+ final ClusterDetailsVO clusterDetails = clusterDetailsDao.findDetail(clusterId, OOBM_ENABLED_DETAIL);
+ if (clusterDetails != null && !Strings.isNullOrEmpty(clusterDetails.getValue()) && !Boolean.valueOf(clusterDetails.getValue())) {
+ return false;
+ }
+ return true;
+ }
+
+ private boolean isOutOfBandManagementEnabledForHost(Long hostId) {
+ if (hostId == null) {
+ return false;
+ }
+ final OutOfBandManagement outOfBandManagementConfig = outOfBandManagementDao.findByHost(hostId);
+ if (outOfBandManagementConfig == null || !outOfBandManagementConfig.isEnabled()) {
+ return false;
+ }
+ return true;
+ }
+
+ private void checkOutOfBandManagementEnabledByZoneClusterHost(final Host host) {
+ if (!isOutOfBandManagementEnabledForZone(host.getDataCenterId())) {
+ throw new CloudRuntimeException("Out-of-band management is disabled for the host's zone. Aborting Operation.");
+ }
+ if (!isOutOfBandManagementEnabledForCluster(host.getClusterId())) {
+ throw new CloudRuntimeException("Out-of-band management is disabled for the host's cluster. Aborting Operation.");
+ }
+ if (!isOutOfBandManagementEnabledForHost(host.getId())) {
+ throw new CloudRuntimeException("Out-of-band management is disabled or not configured for the host. Aborting Operation.");
+ }
+ }
+
+ public boolean isOutOfBandManagementEnabled(final Host host) {
+ return isOutOfBandManagementEnabledForZone(host.getDataCenterId())
+ && isOutOfBandManagementEnabledForCluster(host.getClusterId())
+ && isOutOfBandManagementEnabledForHost(host.getId());
+ }
+
+ public boolean transitionPowerStateToDisabled(List<? extends Host> hosts) {
+ boolean result = true;
+ for (Host host : hosts) {
+ result = result && transitionPowerState(OutOfBandManagement.PowerState.Event.Disabled,
+ outOfBandManagementDao.findByHost(host.getId()));
+ }
+ return result;
+ }
+
+ public void submitBackgroundPowerSyncTask(final Host host) {
+ if (host != null) {
+ // Note: This is a blocking queue based executor
+ backgroundSyncExecutor.submit(new OutOfBandManagementBackgroundTask(this, host, OutOfBandManagement.PowerOperation.STATUS));
+ }
+ }
+
+ private OutOfBandManagementResponse buildEnableDisableResponse(final boolean enabled) {
+ final OutOfBandManagementResponse response = new OutOfBandManagementResponse();
+ response.setEnabled(enabled);
+ response.setSuccess(true);
+ return response;
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_ENABLEDISABLE, eventDescription = "enabling/disabling out-of-band management on a zone")
+ public OutOfBandManagementResponse enableDisableOutOfBandManagement(final DataCenter zone, final boolean enabled) {
+ dataCenterDetailsDao.persist(zone.getId(), OOBM_ENABLED_DETAIL, String.valueOf(enabled));
+ if (!enabled) {
+ transitionPowerStateToDisabled(hostDao.findByDataCenterId(zone.getId()));
+ }
+ return buildEnableDisableResponse(enabled);
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_ENABLEDISABLE, eventDescription = "enabling/disabling out-of-band management on a cluster")
+ public OutOfBandManagementResponse enableDisableOutOfBandManagement(final Cluster cluster, final boolean enabled) {
+ clusterDetailsDao.persist(cluster.getId(), OOBM_ENABLED_DETAIL, String.valueOf(enabled));
+ if (!enabled) {
+ transitionPowerStateToDisabled(hostDao.findByClusterId(cluster.getId()));
+ }
+ return buildEnableDisableResponse(enabled);
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_ENABLEDISABLE, eventDescription = "enabling/disabling out-of-band management on a host")
+ public OutOfBandManagementResponse enableDisableOutOfBandManagement(final Host host, final boolean enabled) {
+ final OutOfBandManagement outOfBandManagementConfig = outOfBandManagementDao.findByHost(host.getId());
+ if (outOfBandManagementConfig == null) {
+ final OutOfBandManagementResponse response = new OutOfBandManagementResponse(null);
+ response.setSuccess(false);
+ response.setResultDescription("Out-of-band management is not configured for the host. Please configure the host before enabling/disabling it.");
+ return response;
+ }
+ hostAlertCache.invalidate(host.getId());
+ outOfBandManagementConfig.setEnabled(enabled);
+ boolean updateResult = outOfBandManagementDao.update(outOfBandManagementConfig.getId(), (OutOfBandManagementVO) outOfBandManagementConfig);
+ if (updateResult && !enabled) {
+ transitionPowerStateToDisabled(Collections.singletonList(host));
+ }
+ return buildEnableDisableResponse(enabled && updateResult);
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_CONFIGURE, eventDescription = "updating out-of-band management configuration")
+ public OutOfBandManagementResponse configureOutOfBandManagement(final Host host, final ImmutableMap<OutOfBandManagement.Option, String> options) {
+ OutOfBandManagement outOfBandManagementConfig = outOfBandManagementDao.findByHost(host.getId());
+ if (outOfBandManagementConfig == null) {
+ outOfBandManagementConfig = outOfBandManagementDao.persist(new OutOfBandManagementVO(host.getId()));
+ }
+ outOfBandManagementConfig = updateConfig(outOfBandManagementConfig, options);
+ if (Strings.isNullOrEmpty(outOfBandManagementConfig.getDriver()) || !outOfBandManagementDriversMap.containsKey(outOfBandManagementConfig.getDriver().toLowerCase())) {
+ throw new CloudRuntimeException("Out-of-band management driver is not available. Please provide a valid driver name.");
+ }
+
+ boolean updatedConfig = outOfBandManagementDao.update(outOfBandManagementConfig.getId(), (OutOfBandManagementVO) outOfBandManagementConfig);
+ CallContext.current().setEventDetails("host id:" + host.getId() + " configuration:" + outOfBandManagementConfig.getAddress() + ":" + outOfBandManagementConfig.getPort());
+
+ if (!updatedConfig) {
+ throw new CloudRuntimeException("Failed to update out-of-band management config for the host in the database.");
+ }
+
+ String result = "Out-of-band management successfully configured for the host";
+ LOG.debug(result);
+
+ final OutOfBandManagementResponse response = new OutOfBandManagementResponse(outOfBandManagementDao.findByHost(host.getId()));
+ response.setResultDescription(result);
+ response.setSuccess(true);
+ return response;
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_ACTION, eventDescription = "issuing Host out-of-band management action", async = true)
+ public OutOfBandManagementResponse executeOutOfBandManagementPowerOperation(final Host host, final OutOfBandManagement.PowerOperation powerOperation, final Long timeout) {
+ checkOutOfBandManagementEnabledByZoneClusterHost(host);
+ final OutOfBandManagement outOfBandManagementConfig = outOfBandManagementDao.findByHost(host.getId());
+ final ImmutableMap<OutOfBandManagement.Option, String> options = getOptions(outOfBandManagementConfig);
+ final OutOfBandManagementDriver driver = getDriver(outOfBandManagementConfig);
+
+ Long actionTimeOut = timeout;
+ if (actionTimeOut == null) {
+ actionTimeOut = OutOfBandManagementActionTimeout.valueIn(host.getClusterId());
+ }
+
+ final OutOfBandManagementDriverPowerCommand cmd = new OutOfBandManagementDriverPowerCommand(options, actionTimeOut, powerOperation);
+ final OutOfBandManagementDriverResponse driverResponse = driver.execute(cmd);
+
+ if (driverResponse == null) {
+ throw new CloudRuntimeException(String.format("Out-of-band Management action (%s) on host (%s) failed due to no response from the driver", powerOperation, host.getUuid()));
+ }
+
+ if (powerOperation.equals(OutOfBandManagement.PowerOperation.STATUS)) {
+ transitionPowerState(driverResponse.toEvent(), outOfBandManagementConfig);
+ }
+
+ if (!driverResponse.isSuccess()) {
+ String errorMessage = String.format("Out-of-band Management action (%s) on host (%s) failed with error: %s", powerOperation, host.getUuid(), driverResponse.getError());
+ if (driverResponse.hasAuthFailure()) {
+ errorMessage = String.format("Out-of-band Management action (%s) on host (%s) failed due to authentication error: %s. Please check configured credentials.", powerOperation, host.getUuid(), driverResponse.getError());
+ sendAuthError(host, errorMessage);
+ }
+ if (!powerOperation.equals(OutOfBandManagement.PowerOperation.STATUS)) {
+ LOG.debug(errorMessage);
+ }
+ throw new CloudRuntimeException(errorMessage);
+ }
+
+ final OutOfBandManagementResponse response = new OutOfBandManagementResponse(outOfBandManagementDao.findByHost(host.getId()));
+ response.setSuccess(driverResponse.isSuccess());
+ response.setResultDescription(driverResponse.getResult());
+ response.setId(host.getUuid());
+ response.setOutOfBandManagementAction(powerOperation.toString());
+ return response;
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_CHANGE_PASSWORD, eventDescription = "updating out-of-band management password")
+ public OutOfBandManagementResponse changeOutOfBandManagementPassword(final Host host, final String newPassword) {
+ checkOutOfBandManagementEnabledByZoneClusterHost(host);
+ if (Strings.isNullOrEmpty(newPassword)) {
+ throw new CloudRuntimeException(String.format("Cannot change out-of-band management password as provided new-password is null or empty for the host %s.", host.getUuid()));
+ }
+ GlobalLock outOfBandManagementHostLock = GlobalLock.getInternLock(getOutOfBandManagementHostLock(host.getId()));
+ try {
+ if (outOfBandManagementHostLock.lock(ACQUIRE_GLOBAL_LOCK_TIMEOUT_FOR_HOST)) {
+ try {
+ final OutOfBandManagement outOfBandManagementConfig = outOfBandManagementDao.findByHost(host.getId());
+
+ final ImmutableMap<OutOfBandManagement.Option, String> options = getOptions(outOfBandManagementConfig);
+ if (!(options.containsKey(OutOfBandManagement.Option.PASSWORD) && !Strings.isNullOrEmpty(options.get(OutOfBandManagement.Option.PASSWORD)))) {
+ throw new CloudRuntimeException(String.format("Cannot change out-of-band management password as we've no previously configured password for the host %s.", host.getUuid()));
+ }
+ final OutOfBandManagementDriver driver = getDriver(outOfBandManagementConfig);
+
+ final OutOfBandManagementDriverChangePasswordCommand cmd = new OutOfBandManagementDriverChangePasswordCommand(options, OutOfBandManagementActionTimeout.valueIn(host.getClusterId()), newPassword);
+ final OutOfBandManagementDriverResponse driverResponse;
+ try {
+ driverResponse = driver.execute(cmd);
+ } catch (Exception e) {
+ LOG.error("Out-of-band management change password failed due to driver error: " + e.getMessage());
+ throw new CloudRuntimeException(String.format("Failed to change out-of-band management password for host (%s) due to driver error: %s", host.getUuid(), e.getMessage()));
+ }
+
+ if (!driverResponse.isSuccess()) {
+ throw new CloudRuntimeException(String.format("Failed to change out-of-band management password for host (%s) with error: %s", host.getUuid(), driverResponse.getError()));
+ }
+
+ final boolean updatedConfigResult = Transaction.execute(new TransactionCallback<Boolean>() {
+ @Override
+ public Boolean doInTransaction(TransactionStatus status) {
+ OutOfBandManagement updatedOutOfBandManagementConfig = outOfBandManagementDao.findByHost(host.getId());
+ updatedOutOfBandManagementConfig.setPassword(newPassword);
+ return outOfBandManagementDao.update(updatedOutOfBandManagementConfig.getId(), (OutOfBandManagementVO) updatedOutOfBandManagementConfig);
+ }
+ });
+
+ if (!updatedConfigResult) {
+ LOG.error(String.format("Succeeded to change out-of-band management password but failed to updated in database the new password:%s for the host id:%d", newPassword, host.getId()));
+ }
+
+ final OutOfBandManagementResponse response = new OutOfBandManagementResponse();
+ response.setSuccess(updatedConfigResult && driverResponse.isSuccess());
+ response.setResultDescription(driverResponse.getResult());
+ response.setId(host.getUuid());
+ return response;
+ } finally {
+ outOfBandManagementHostLock.unlock();
+ }
+ } else {
+ LOG.error("Unable to acquire synchronization lock to change out-of-band management password for host id: " + host.getId());
+ throw new CloudRuntimeException(String.format("Unable to acquire lock to change out-of-band management password for host (%s), please try after some time.", host.getUuid()));
+ }
+ } finally {
+ outOfBandManagementHostLock.releaseRef();
+ }
+ }
+
+ @Override
+ public String getName() {
+ return name;
+ }
+
+ @Override
+ public long getId() {
+ return serviceId;
+ }
+
+ @Override
+ public boolean configure(final String name, final Map<String, Object> params) throws ConfigurationException {
+ this.name = name;
+ this.serviceId = ManagementServerNode.getManagementServerId();
+
+ final int poolSize = OutOfBandManagementSyncThreadPoolSize.value();
+
+ hostAlertCache = CacheBuilder.newBuilder()
+ .concurrencyLevel(4)
+ .weakKeys()
+ .maximumSize(100 * poolSize)
+ .expireAfterWrite(1, TimeUnit.DAYS)
+ .build();
+
+ backgroundSyncExecutor = new ThreadPoolExecutor(poolSize, poolSize,
+ 0L, TimeUnit.MILLISECONDS,
+ new ArrayBlockingQueue<Runnable>(10 * poolSize, true), new ThreadPoolExecutor.CallerRunsPolicy());
+
+ LOG.info("Starting out-of-band management background sync executor with thread pool-size=" + poolSize + " and background sync thread interval=" + OutOfBandManagementSyncThreadInterval.value() + "s");
+ return true;
+ }
+
+ @Override
+ public boolean start() {
+ initializeDriversMap();
+ return true;
+ }
+
+ @Override
+ public boolean stop() {
+ backgroundSyncExecutor.shutdown();
+ outOfBandManagementDao.expireOutOfBandManagementOwnershipByServer(getId());
+ return true;
+ }
+
+ @Override
+ public String getConfigComponentName() {
+ return OutOfBandManagementServiceImpl.class.getSimpleName();
+ }
+
+ @Override
+ public ConfigKey<?>[] getConfigKeys() {
+ return new ConfigKey<?>[] {OutOfBandManagementActionTimeout, OutOfBandManagementSyncThreadInterval, OutOfBandManagementSyncThreadPoolSize};
+ }
+
+ public List<OutOfBandManagementDriver> getOutOfBandManagementDrivers() {
+ return outOfBandManagementDrivers;
+ }
+
+ public void setOutOfBandManagementDrivers(List<OutOfBandManagementDriver> outOfBandManagementDrivers) {
+ this.outOfBandManagementDrivers = outOfBandManagementDrivers;
--- End diff --
This is spring injected, we need the same object that spring injected to this class.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61779846
--- Diff: api/src/org/apache/cloudstack/outofbandmanagement/driver/OutOfBandManagementDriverPowerCommand.java ---
@@ -0,0 +1,33 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.outofbandmanagement.driver;
+
+import com.google.common.collect.ImmutableMap;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagement;
+
+public class OutOfBandManagementDriverPowerCommand extends OutOfBandManagementDriverCommand {
--- End diff --
Why not declare this class as ``final``? If it is possible, then no methods need to be marked as such.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by bhaisaab <gi...@git.apache.org>.
Github user bhaisaab commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r60716885
--- Diff: client/tomcatconf/commands.properties.in ---
@@ -799,3 +799,14 @@ quotaCredits=1
quotaEmailTemplateList=1
quotaEmailTemplateUpdate=1
quotaIsEnabled=15
+
+### Out-of-band Management
--- End diff --
If rbac gets merged, we won't need changes in this file as well
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by swill <gi...@git.apache.org>.
Github user swill commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-213421943
Thanks guys. I will try to get this one queued up for CI.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by bhaisaab <gi...@git.apache.org>.
Github user bhaisaab commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-213375165
Pinging for review @jburwell @swill @wido @pyr @koushik-das @agneya2001 @DaanHoogland @rafaelweingartner @GabrielBrascher @kishankavala
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61843031
--- Diff: api/src/org/apache/cloudstack/api/command/admin/outofbandmanagement/EnableOutOfBandManagementForHostCmd.java ---
@@ -0,0 +1,107 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.api.command.admin.outofbandmanagement;
+
+import com.cloud.event.EventTypes;
+import com.cloud.exception.ConcurrentOperationException;
+import com.cloud.exception.InsufficientCapacityException;
+import com.cloud.exception.NetworkRuleConflictException;
+import com.cloud.exception.ResourceAllocationException;
+import com.cloud.exception.ResourceUnavailableException;
+import com.cloud.host.Host;
+import org.apache.cloudstack.acl.RoleType;
+import org.apache.cloudstack.api.APICommand;
+import org.apache.cloudstack.api.ApiConstants;
+import org.apache.cloudstack.api.ApiErrorCode;
+import org.apache.cloudstack.api.BaseAsyncCmd;
+import org.apache.cloudstack.api.BaseCmd;
+import org.apache.cloudstack.api.Parameter;
+import org.apache.cloudstack.api.ServerApiException;
+import org.apache.cloudstack.api.response.HostResponse;
+import org.apache.cloudstack.api.response.OutOfBandManagementResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagementService;
+
+import javax.inject.Inject;
+
+@APICommand(name = "enableOutOfBandManagementForHost", description = "Enables out-of-band management for a host",
--- End diff --
fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by pyr <gi...@git.apache.org>.
Github user pyr commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-213593763
First read-through didn't raise any eyebrows on my end. LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by pdion891 <gi...@git.apache.org>.
Github user pdion891 commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-214824625
@swill,what are those changes you are refering too? are they required for regular deployment or just for tests? anyway we will need some info for the release notes has it seams like very cool new feature!
look like we will need documentation for the admin guide too which is almost done from this PR comments!Cool!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by wido <gi...@git.apache.org>.
Github user wido commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r60825819
--- Diff: api/src/org/apache/cloudstack/api/command/admin/outofbandmanagement/ConfigureOutOfBandManagementCmd.java ---
@@ -0,0 +1,127 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.api.command.admin.outofbandmanagement;
+
+import com.cloud.exception.ConcurrentOperationException;
+import com.cloud.exception.InsufficientCapacityException;
+import com.cloud.exception.NetworkRuleConflictException;
+import com.cloud.exception.ResourceAllocationException;
+import com.cloud.exception.ResourceUnavailableException;
+import com.cloud.host.Host;
+import com.google.common.base.Strings;
+import com.google.common.collect.ImmutableMap;
+import org.apache.cloudstack.acl.RoleType;
+import org.apache.cloudstack.api.APICommand;
+import org.apache.cloudstack.api.ApiConstants;
+import org.apache.cloudstack.api.ApiErrorCode;
+import org.apache.cloudstack.api.BaseCmd;
+import org.apache.cloudstack.api.Parameter;
+import org.apache.cloudstack.api.ServerApiException;
+import org.apache.cloudstack.api.response.HostResponse;
+import org.apache.cloudstack.api.response.OutOfBandManagementResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagement;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagementService;
+
+import javax.inject.Inject;
+
+@APICommand(name = "configureOutOfBandManagement", description = "Configures a host's out-of-band management interface",
+ responseObject = OutOfBandManagementResponse.class, requestHasSensitiveInfo = true, responseHasSensitiveInfo = false, authorized = {RoleType.Admin})
+public class ConfigureOutOfBandManagementCmd extends BaseCmd {
+ @Inject
+ private OutOfBandManagementService outOfBandManagementService;
+
+ /////////////////////////////////////////////////////
+ //////////////// API parameters /////////////////////
+ /////////////////////////////////////////////////////
+
+ @Parameter(name = ApiConstants.HOST_ID, type = CommandType.UUID, entityType = HostResponse.class, required = true, description = "the ID of the host")
+ private Long hostId;
+
+ @Parameter(name = ApiConstants.DRIVER, type = CommandType.STRING, required = true, description = "the host management interface driver, for example: ipmitool")
+ private String driver;
+
+ @Parameter(name = ApiConstants.IP_ADDRESS, type = CommandType.STRING, required = true, description = "the host management interface IP address")
--- End diff --
Why is a IP-Address a String? Java has a native InetAddress for this
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61816081
--- Diff: test/integration/smoke/test_outofbandmanagement.py ---
@@ -0,0 +1,561 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import marvin
+from marvin.cloudstackTestCase import *
+from marvin.cloudstackAPI import *
+from marvin.lib.utils import *
+from marvin.lib.base import *
+from marvin.lib.common import *
+from marvin.lib.utils import (random_gen)
+from nose.plugins.attrib import attr
+
+from ipmisim.ipmisim import IpmiServerContext, IpmiServer, ThreadedIpmiServer
+
+import socket
+import sys
+import thread
+import time
+
+
+class TestOutOfBandManagement(cloudstackTestCase):
+ """ Test cases for out of band management
+ """
+
+ def setUp(self):
+ self.apiclient = self.testClient.getApiClient()
+ self.hypervisor = self.testClient.getHypervisorInfo()
+ self.dbclient = self.testClient.getDbConnection()
+ self.services = self.testClient.getParsedTestDataConfig()
+ self.mgtSvrDetails = self.config.__dict__["mgtSvr"][0].__dict__
+
+ self.zone = get_zone(self.apiclient, self.testClient.getZoneForTests())
+ self.host = None
+ self.server = None
+
+ # use random port for ipmisim
+ s = socket.socket()
+ s.bind(('', 0))
+ self.serverPort = s.getsockname()[1]
+ s.close()
+
+ self.cleanup = []
+
+
+ def tearDown(self):
+ try:
+ self.dbclient.execute("delete from oobm where port=%d" % self.getIpmiServerPort())
+ self.dbclient.execute("delete from mshost_peer where peer_runid=%s" % self.getFakeMsRunId())
+ self.dbclient.execute("delete from mshost where runid=%s" % self.getFakeMsRunId())
+ self.dbclient.execute("delete from cluster_details where name='outOfBandManagementEnabled'")
+ self.dbclient.execute("delete from data_center_details where name='outOfBandManagementEnabled'")
+ cleanup_resources(self.apiclient, self.cleanup)
+ if self.server:
+ self.server.shutdown()
+ self.server.server_close()
+ except Exception as e:
+ raise Exception("Warning: Exception during cleanup : %s" % e)
+
+
+ def getFakeMsId(self):
+ return 1234567890
+
+
+ def getFakeMsRunId(self):
+ return 123456
+
+
+ def getHost(self, hostId=None):
+ if self.host and hostId is None:
+ return self.host
+
+ response = list_hosts(
+ self.apiclient,
+ zoneid=self.zone.id,
+ type='Routing',
+ id=hostId
+ )
+ if len(response) > 0:
+ self.host = response[0]
+ return self.host
+ raise self.skipTest("No hosts found, skipping out-of-band management test")
+
+
+ def getIpmiServerIp(self):
+ s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
+ s.connect((self.mgtSvrDetails["mgtSvrIp"], self.mgtSvrDetails["port"]))
+ return s.getsockname()[0]
+
+
+ def getIpmiServerPort(self):
+ return self.serverPort
+
+
+ def getOobmConfigCmd(self):
+ cmd = configureOutOfBandManagement.configureOutOfBandManagementCmd()
+ cmd.driver = 'ipmitool' # The default available driver
+ cmd.address = self.getIpmiServerIp()
+ cmd.port = self.getIpmiServerPort()
+ cmd.username = 'admin'
+ cmd.password = 'password'
+ cmd.hostid = self.getHost().id
+ return cmd
+
+
+ def getOobmEnableCmd(self):
+ cmd = enableOutOfBandManagementForHost.enableOutOfBandManagementForHostCmd()
+ cmd.hostid = self.getHost().id
+ return cmd
+
+
+ def getOobmDisableCmd(self):
+ cmd = disableOutOfBandManagementForHost.disableOutOfBandManagementForHostCmd()
+ cmd.hostid = self.getHost().id
+ return cmd
+
+
+ def getOobmIssueActionCmd(self):
+ cmd = issueOutOfBandManagementPowerAction.issueOutOfBandManagementPowerActionCmd()
+ cmd.hostid = self.getHost().id
+ cmd.action = 'STATUS'
+ return cmd
+
+
+ def issuePowerActionCmd(self, action, timeout=None):
+ cmd = self.getOobmIssueActionCmd()
+ cmd.action = action
+ if timeout:
+ cmd.timeout = timeout
+ return self.apiclient.issueOutOfBandManagementPowerAction(cmd)
+
+
+ def configureAndEnableOobm(self):
+ self.apiclient.configureOutOfBandManagement(self.getOobmConfigCmd())
+ response = self.apiclient.enableOutOfBandManagementForHost(self.getOobmEnableCmd())
+ self.assertEqual(response.enabled, True)
+
+
+ def startIpmiServer(self):
+ def startIpmiServer(tname, server):
+ self.debug("Starting ipmisim server")
+ try:
+ server.serve_forever()
+ except Exception: pass
+ IpmiServerContext('reset')
+ ThreadedIpmiServer.allow_reuse_address = False
+ server = ThreadedIpmiServer(('0.0.0.0', self.getIpmiServerPort()), IpmiServer)
+ thread.start_new_thread(startIpmiServer, ("ipmi-server", server,))
+ self.server = server
+
+
+ def checkSyncToState(self, state, interval):
+ self.debug("Waiting for background thread to update powerstate to " + state)
+ time.sleep(1 + int(interval)*2/1000) # interval is in ms
+ response = self.getHost(hostId=self.getHost().id).outofbandmanagement
+ self.assertEqual(response.powerstate, state)
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_configure_invalid_driver(self):
+ """
+ Tests out-of-band management configuration with invalid driver
+ """
+ cmd = self.getOobmConfigCmd()
+ cmd.driver = 'randomDriverThatDoesNotExist'
+ try:
+ response = self.apiclient.configureOutOfBandManagement(cmd)
+ self.fail("Expected an exception to be thrown, failing")
+ except Exception: pass
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_configure_default_driver(self):
+ """
+ Tests out-of-band management configuration with valid data
+ """
+ cmd = self.getOobmConfigCmd()
+ response = self.apiclient.configureOutOfBandManagement(cmd)
+ self.assertEqual(response.hostid, cmd.hostid)
+ self.assertEqual(response.driver, cmd.driver)
+ self.assertEqual(response.address, cmd.address)
+ self.assertEqual(response.port, str(cmd.port))
+ self.assertEqual(response.username, cmd.username)
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_enable_feature_invalid(self):
+ """
+ Tests out-of-band management host enable feature with
+ invalid options
+ """
+ cmd = self.getOobmEnableCmd()
+ cmd.hostid = -1
+ try:
+ response = self.apiclient.enableOutOfBandManagementForHost(cmd)
+ self.fail("Expected an exception to be thrown, failing")
+ except Exception: pass
+
+ try:
+ cmd = enableOutOfBandManagementForCluster.enableOutOfBandManagementForClusterCmd()
+ response = self.apiclient.enableOutOfBandManagementForCluster(cmd)
+ self.fail("Expected an exception to be thrown, failing")
+ except Exception: pass
+
+ try:
+ cmd = enableOutOfBandManagementForZone.enableOutOfBandManagementForZoneCmd()
+ response = self.apiclient.enableOutOfBandManagementForZone(cmd)
+ self.fail("Expected an exception to be thrown, failing")
+ except Exception: pass
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_disable_feature_invalid(self):
+ """
+ Tests out-of-band management host disable feature with
+ invalid options
+ """
+ cmd = self.getOobmDisableCmd()
+ cmd.hostid = -1
+ try:
+ response = self.apiclient.disableOutOfBandManagementForHost(cmd)
+ self.fail("Expected an exception to be thrown, failing")
+ except Exception: pass
+
+ try:
+ cmd = disableOutOfBandManagementForCluster.disableOutOfBandManagementForClusterCmd()
+ response = self.apiclient.disableOutOfBandManagementForCluster(cmd)
+ self.fail("Expected an exception to be thrown, failing")
+ except Exception: pass
+
+ try:
+ cmd = disableOutOfBandManagementForZone.disableOutOfBandManagementForZoneCmd()
+ response = self.apiclient.disableOutOfBandManagementForZone(cmd)
+ self.fail("Expected an exception to be thrown, failing")
+ except Exception: pass
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_enable_feature_valid(self):
+ """
+ Tests out-of-band management host enable feature with
+ valid options
+ """
+ self.apiclient.configureOutOfBandManagement(self.getOobmConfigCmd())
+ cmd = self.getOobmEnableCmd()
+ response = self.apiclient.enableOutOfBandManagementForHost(cmd)
+ self.assertEqual(response.hostid, cmd.hostid)
+ self.assertEqual(response.enabled, True)
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_disable_feature_valid(self):
+ """
+ Tests out-of-band management host disable feature with
+ valid options
+ """
+
+ self.apiclient.configureOutOfBandManagement(self.getOobmConfigCmd())
+ cmd = self.getOobmDisableCmd()
+ response = self.apiclient.disableOutOfBandManagementForHost(cmd)
+ self.assertEqual(response.hostid, cmd.hostid)
+ self.assertEqual(response.enabled, False)
+
+ response = self.getHost(hostId=cmd.hostid).outofbandmanagement
+ self.assertEqual(response.powerstate, 'Disabled')
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_enabledisable_across_clusterzones(self):
--- End diff --
Could this method be separated for zone, cluster, and host? It seems like it would make the test case more readable.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61809934
--- Diff: server/src/org/apache/cloudstack/outofbandmanagement/OutOfBandManagementServiceImpl.java ---
@@ -0,0 +1,532 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.outofbandmanagement;
+
+import com.cloud.alert.AlertManager;
+import com.cloud.dc.ClusterDetailsDao;
+import com.cloud.dc.ClusterDetailsVO;
+import com.cloud.dc.DataCenter;
+import com.cloud.dc.DataCenterDetailVO;
+import com.cloud.dc.dao.DataCenterDetailsDao;
+import com.cloud.domain.Domain;
+import com.cloud.event.ActionEvent;
+import com.cloud.event.ActionEventUtils;
+import com.cloud.event.EventTypes;
+import com.cloud.host.Host;
+import com.cloud.host.dao.HostDao;
+import com.cloud.org.Cluster;
+import com.cloud.utils.component.Manager;
+import com.cloud.utils.component.ManagerBase;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.db.Transaction;
+import com.cloud.utils.db.TransactionCallback;
+import com.cloud.utils.db.TransactionStatus;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.fsm.NoTransitionException;
+import com.google.common.base.Strings;
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import com.google.common.collect.ImmutableMap;
+import org.apache.cloudstack.api.response.OutOfBandManagementResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.framework.config.ConfigKey;
+import org.apache.cloudstack.framework.config.Configurable;
+import org.apache.cloudstack.outofbandmanagement.dao.OutOfBandManagementDao;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverChangePasswordCommand;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverPowerCommand;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverResponse;
+import org.apache.cloudstack.utils.identity.ManagementServerNode;
+import org.apache.log4j.Logger;
+import org.springframework.stereotype.Component;
+
+import javax.ejb.Local;
+import javax.inject.Inject;
+import javax.naming.ConfigurationException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+@Component
+@Local(value = {OutOfBandManagementService.class})
+public class OutOfBandManagementServiceImpl extends ManagerBase implements OutOfBandManagementService, Manager, Configurable {
+ public static final Logger LOG = Logger.getLogger(OutOfBandManagementServiceImpl.class);
+
+ @Inject
+ private ClusterDetailsDao clusterDetailsDao;
+ @Inject
+ private DataCenterDetailsDao dataCenterDetailsDao;
+ @Inject
+ private OutOfBandManagementDao outOfBandManagementDao;
+ @Inject
+ private HostDao hostDao;
+ @Inject
+ private AlertManager alertMgr;
+
+ private String name;
+ private long serviceId;
+
+ private List<OutOfBandManagementDriver> outOfBandManagementDrivers = new ArrayList<>();
+ private Map<String, OutOfBandManagementDriver> outOfBandManagementDriversMap = new HashMap<String, OutOfBandManagementDriver>();
+
+ private static final String OOBM_ENABLED_DETAIL = "outOfBandManagementEnabled";
+ private static final int ACQUIRE_GLOBAL_LOCK_TIMEOUT_FOR_HOST = 120;
+
+ private Cache<Long, Long> hostAlertCache;
+ private static ExecutorService backgroundSyncExecutor;
+
+ private String getOutOfBandManagementHostLock(long id) {
+ return "oobm.host." + id;
+ }
+
+ private void initializeDriversMap() {
+ if (outOfBandManagementDriversMap.isEmpty() && outOfBandManagementDrivers != null && outOfBandManagementDrivers.size() > 0) {
+ for (final OutOfBandManagementDriver driver : outOfBandManagementDrivers) {
+ outOfBandManagementDriversMap.put(driver.getName().toLowerCase(), driver);
+ }
+ LOG.debug("Discovered out-of-band management drivers configured in the OutOfBandManagementService");
+ }
+ }
+
+ private OutOfBandManagementDriver getDriver(final OutOfBandManagement outOfBandManagementConfig) {
+ if (!Strings.isNullOrEmpty(outOfBandManagementConfig.getDriver())) {
+ final OutOfBandManagementDriver driver = outOfBandManagementDriversMap.get(outOfBandManagementConfig.getDriver());
+ if (driver != null) {
+ return driver;
+ }
+ }
+ throw new CloudRuntimeException("Configured out-of-band management driver is not available. Aborting any out-of-band management action.");
+ }
+
+ protected OutOfBandManagement updateConfig(final OutOfBandManagement outOfBandManagementConfig, final ImmutableMap<OutOfBandManagement.Option, String> options) {
+ if (outOfBandManagementConfig == null) {
+ throw new CloudRuntimeException("Out-of-band management is not configured for the host. Aborting.");
+ }
+ if (options == null) {
+ return outOfBandManagementConfig;
+ }
+ for (OutOfBandManagement.Option option: options.keySet()) {
+ final String value = options.get(option);
+ if (Strings.isNullOrEmpty(value)) {
+ continue;
+ }
+ switch (option) {
+ case DRIVER:
+ outOfBandManagementConfig.setDriver(value);
+ break;
+ case ADDRESS:
+ outOfBandManagementConfig.setAddress(value);
+ break;
+ case PORT:
+ outOfBandManagementConfig.setPort(Integer.parseInt(value));
+ break;
+ case USERNAME:
+ outOfBandManagementConfig.setUsername(value);
+ break;
+ case PASSWORD:
+ outOfBandManagementConfig.setPassword(value);
+ break;
+ }
+ }
+ return outOfBandManagementConfig;
+ }
+
+ protected ImmutableMap<OutOfBandManagement.Option, String> getOptions(final OutOfBandManagement outOfBandManagementConfig) {
+ final ImmutableMap.Builder<OutOfBandManagement.Option, String> optionsBuilder = ImmutableMap.builder();
+ if (outOfBandManagementConfig == null) {
+ throw new CloudRuntimeException("Out-of-band management is not configured for the host. Aborting.");
+ }
+ for (OutOfBandManagement.Option option: OutOfBandManagement.Option.values()) {
+ String value = null;
+ switch (option) {
+ case DRIVER:
+ value = outOfBandManagementConfig.getDriver();
+ break;
+ case ADDRESS:
+ value = outOfBandManagementConfig.getAddress();
+ break;
+ case PORT:
+ if (outOfBandManagementConfig.getPort() != null) {
+ value = String.valueOf(outOfBandManagementConfig.getPort());
+ }
+ break;
+ case USERNAME:
+ value = outOfBandManagementConfig.getUsername();
+ break;
+ case PASSWORD:
+ value = outOfBandManagementConfig.getPassword();
+ break;
+ }
+ if (value != null) {
+ optionsBuilder.put(option, value);
+ }
+ }
+ return optionsBuilder.build();
+ }
+
+ private void sendAuthError(final Host host, final String message) {
+ try {
+ hostAlertCache.asMap().putIfAbsent(host.getId(), 0L);
+ Long sentCount = hostAlertCache.asMap().get(host.getId());
+ if (sentCount != null && sentCount <= 0) {
+ boolean concurrentUpdateResult = hostAlertCache.asMap().replace(host.getId(), sentCount, sentCount+1L);
+ if (concurrentUpdateResult) {
+ final String subject = String.format("Out-of-band management auth-error detected for host:%d in cluster:%d, zone:%d", host.getId(), host.getClusterId(), host.getDataCenterId());
+ LOG.error(subject + ": " + message);
+ alertMgr.sendAlert(AlertManager.AlertType.ALERT_TYPE_OOBM_AUTH_ERROR, host.getDataCenterId(), host.getPodId(), subject, message);
+ }
+ }
+ } catch (Exception ignored) {
+ }
+ }
+
+ private boolean transitionPowerState(OutOfBandManagement.PowerState.Event event, OutOfBandManagement outOfBandManagementHost) {
+ if (outOfBandManagementHost == null) {
+ return false;
+ }
+ OutOfBandManagement.PowerState currentPowerState = outOfBandManagementHost.getPowerState();
+ try {
+ OutOfBandManagement.PowerState newPowerState = OutOfBandManagement.PowerState.getStateMachine().getNextState(currentPowerState, event);
+ boolean result = outOfBandManagementDao.updateState(currentPowerState, event, newPowerState, outOfBandManagementHost, null);
+ if (result) {
+ final String message = String.format("Transitioned out-of-band management power state from:%s to:%s due to event:%s for the host id:%d", currentPowerState, newPowerState, event, outOfBandManagementHost.getHostId());
+ LOG.debug(message);
+ ActionEventUtils.onActionEvent(CallContext.current().getCallingUserId(), CallContext.current().getCallingAccountId(), Domain.ROOT_DOMAIN,
+ EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_POWERSTATE_TRANSITION, message);
+ }
+ return result;
+ } catch (NoTransitionException ignored) {
+ LOG.trace(String.format("Unable to transition out-of-band management power state for host id=%s for the event=%s and current power state=%s", outOfBandManagementHost.getHostId(), event, currentPowerState));
+ }
+ return false;
+ }
+
+ private boolean isOutOfBandManagementEnabledForZone(Long zoneId) {
+ if (zoneId == null) {
+ return true;
+ }
+ final DataCenterDetailVO zoneDetails = dataCenterDetailsDao.findDetail(zoneId, OOBM_ENABLED_DETAIL);
+ if (zoneDetails != null && !Strings.isNullOrEmpty(zoneDetails.getValue()) && !Boolean.valueOf(zoneDetails.getValue())) {
+ return false;
+ }
+ return true;
+ }
+
+ private boolean isOutOfBandManagementEnabledForCluster(Long clusterId) {
+ if (clusterId == null) {
+ return true;
+ }
+ final ClusterDetailsVO clusterDetails = clusterDetailsDao.findDetail(clusterId, OOBM_ENABLED_DETAIL);
+ if (clusterDetails != null && !Strings.isNullOrEmpty(clusterDetails.getValue()) && !Boolean.valueOf(clusterDetails.getValue())) {
+ return false;
+ }
+ return true;
+ }
+
+ private boolean isOutOfBandManagementEnabledForHost(Long hostId) {
+ if (hostId == null) {
+ return false;
+ }
+ final OutOfBandManagement outOfBandManagementConfig = outOfBandManagementDao.findByHost(hostId);
+ if (outOfBandManagementConfig == null || !outOfBandManagementConfig.isEnabled()) {
+ return false;
+ }
+ return true;
+ }
+
+ private void checkOutOfBandManagementEnabledByZoneClusterHost(final Host host) {
+ if (!isOutOfBandManagementEnabledForZone(host.getDataCenterId())) {
+ throw new CloudRuntimeException("Out-of-band management is disabled for the host's zone. Aborting Operation.");
+ }
+ if (!isOutOfBandManagementEnabledForCluster(host.getClusterId())) {
+ throw new CloudRuntimeException("Out-of-band management is disabled for the host's cluster. Aborting Operation.");
+ }
+ if (!isOutOfBandManagementEnabledForHost(host.getId())) {
+ throw new CloudRuntimeException("Out-of-band management is disabled or not configured for the host. Aborting Operation.");
+ }
+ }
+
+ public boolean isOutOfBandManagementEnabled(final Host host) {
+ return isOutOfBandManagementEnabledForZone(host.getDataCenterId())
+ && isOutOfBandManagementEnabledForCluster(host.getClusterId())
+ && isOutOfBandManagementEnabledForHost(host.getId());
+ }
+
+ public boolean transitionPowerStateToDisabled(List<? extends Host> hosts) {
+ boolean result = true;
+ for (Host host : hosts) {
+ result = result && transitionPowerState(OutOfBandManagement.PowerState.Event.Disabled,
+ outOfBandManagementDao.findByHost(host.getId()));
+ }
+ return result;
+ }
+
+ public void submitBackgroundPowerSyncTask(final Host host) {
+ if (host != null) {
+ // Note: This is a blocking queue based executor
+ backgroundSyncExecutor.submit(new OutOfBandManagementBackgroundTask(this, host, OutOfBandManagement.PowerOperation.STATUS));
+ }
+ }
+
+ private OutOfBandManagementResponse buildEnableDisableResponse(final boolean enabled) {
+ final OutOfBandManagementResponse response = new OutOfBandManagementResponse();
+ response.setEnabled(enabled);
+ response.setSuccess(true);
+ return response;
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_ENABLEDISABLE, eventDescription = "enabling/disabling out-of-band management on a zone")
+ public OutOfBandManagementResponse enableDisableOutOfBandManagement(final DataCenter zone, final boolean enabled) {
+ dataCenterDetailsDao.persist(zone.getId(), OOBM_ENABLED_DETAIL, String.valueOf(enabled));
+ if (!enabled) {
+ transitionPowerStateToDisabled(hostDao.findByDataCenterId(zone.getId()));
+ }
+ return buildEnableDisableResponse(enabled);
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_ENABLEDISABLE, eventDescription = "enabling/disabling out-of-band management on a cluster")
+ public OutOfBandManagementResponse enableDisableOutOfBandManagement(final Cluster cluster, final boolean enabled) {
+ clusterDetailsDao.persist(cluster.getId(), OOBM_ENABLED_DETAIL, String.valueOf(enabled));
+ if (!enabled) {
+ transitionPowerStateToDisabled(hostDao.findByClusterId(cluster.getId()));
+ }
+ return buildEnableDisableResponse(enabled);
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_ENABLEDISABLE, eventDescription = "enabling/disabling out-of-band management on a host")
+ public OutOfBandManagementResponse enableDisableOutOfBandManagement(final Host host, final boolean enabled) {
+ final OutOfBandManagement outOfBandManagementConfig = outOfBandManagementDao.findByHost(host.getId());
+ if (outOfBandManagementConfig == null) {
+ final OutOfBandManagementResponse response = new OutOfBandManagementResponse(null);
+ response.setSuccess(false);
+ response.setResultDescription("Out-of-band management is not configured for the host. Please configure the host before enabling/disabling it.");
+ return response;
+ }
+ hostAlertCache.invalidate(host.getId());
+ outOfBandManagementConfig.setEnabled(enabled);
+ boolean updateResult = outOfBandManagementDao.update(outOfBandManagementConfig.getId(), (OutOfBandManagementVO) outOfBandManagementConfig);
+ if (updateResult && !enabled) {
+ transitionPowerStateToDisabled(Collections.singletonList(host));
+ }
+ return buildEnableDisableResponse(enabled && updateResult);
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_CONFIGURE, eventDescription = "updating out-of-band management configuration")
+ public OutOfBandManagementResponse configureOutOfBandManagement(final Host host, final ImmutableMap<OutOfBandManagement.Option, String> options) {
+ OutOfBandManagement outOfBandManagementConfig = outOfBandManagementDao.findByHost(host.getId());
+ if (outOfBandManagementConfig == null) {
+ outOfBandManagementConfig = outOfBandManagementDao.persist(new OutOfBandManagementVO(host.getId()));
+ }
+ outOfBandManagementConfig = updateConfig(outOfBandManagementConfig, options);
+ if (Strings.isNullOrEmpty(outOfBandManagementConfig.getDriver()) || !outOfBandManagementDriversMap.containsKey(outOfBandManagementConfig.getDriver().toLowerCase())) {
+ throw new CloudRuntimeException("Out-of-band management driver is not available. Please provide a valid driver name.");
+ }
+
+ boolean updatedConfig = outOfBandManagementDao.update(outOfBandManagementConfig.getId(), (OutOfBandManagementVO) outOfBandManagementConfig);
+ CallContext.current().setEventDetails("host id:" + host.getId() + " configuration:" + outOfBandManagementConfig.getAddress() + ":" + outOfBandManagementConfig.getPort());
+
+ if (!updatedConfig) {
+ throw new CloudRuntimeException("Failed to update out-of-band management config for the host in the database.");
+ }
+
+ String result = "Out-of-band management successfully configured for the host";
+ LOG.debug(result);
+
+ final OutOfBandManagementResponse response = new OutOfBandManagementResponse(outOfBandManagementDao.findByHost(host.getId()));
+ response.setResultDescription(result);
+ response.setSuccess(true);
+ return response;
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_ACTION, eventDescription = "issuing Host out-of-band management action", async = true)
+ public OutOfBandManagementResponse executeOutOfBandManagementPowerOperation(final Host host, final OutOfBandManagement.PowerOperation powerOperation, final Long timeout) {
+ checkOutOfBandManagementEnabledByZoneClusterHost(host);
+ final OutOfBandManagement outOfBandManagementConfig = outOfBandManagementDao.findByHost(host.getId());
+ final ImmutableMap<OutOfBandManagement.Option, String> options = getOptions(outOfBandManagementConfig);
+ final OutOfBandManagementDriver driver = getDriver(outOfBandManagementConfig);
+
+ Long actionTimeOut = timeout;
+ if (actionTimeOut == null) {
+ actionTimeOut = OutOfBandManagementActionTimeout.valueIn(host.getClusterId());
+ }
+
+ final OutOfBandManagementDriverPowerCommand cmd = new OutOfBandManagementDriverPowerCommand(options, actionTimeOut, powerOperation);
+ final OutOfBandManagementDriverResponse driverResponse = driver.execute(cmd);
+
+ if (driverResponse == null) {
+ throw new CloudRuntimeException(String.format("Out-of-band Management action (%s) on host (%s) failed due to no response from the driver", powerOperation, host.getUuid()));
+ }
+
+ if (powerOperation.equals(OutOfBandManagement.PowerOperation.STATUS)) {
+ transitionPowerState(driverResponse.toEvent(), outOfBandManagementConfig);
+ }
+
+ if (!driverResponse.isSuccess()) {
+ String errorMessage = String.format("Out-of-band Management action (%s) on host (%s) failed with error: %s", powerOperation, host.getUuid(), driverResponse.getError());
+ if (driverResponse.hasAuthFailure()) {
+ errorMessage = String.format("Out-of-band Management action (%s) on host (%s) failed due to authentication error: %s. Please check configured credentials.", powerOperation, host.getUuid(), driverResponse.getError());
+ sendAuthError(host, errorMessage);
+ }
+ if (!powerOperation.equals(OutOfBandManagement.PowerOperation.STATUS)) {
+ LOG.debug(errorMessage);
+ }
+ throw new CloudRuntimeException(errorMessage);
+ }
+
+ final OutOfBandManagementResponse response = new OutOfBandManagementResponse(outOfBandManagementDao.findByHost(host.getId()));
+ response.setSuccess(driverResponse.isSuccess());
+ response.setResultDescription(driverResponse.getResult());
+ response.setId(host.getUuid());
+ response.setOutOfBandManagementAction(powerOperation.toString());
+ return response;
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_CHANGE_PASSWORD, eventDescription = "updating out-of-band management password")
+ public OutOfBandManagementResponse changeOutOfBandManagementPassword(final Host host, final String newPassword) {
+ checkOutOfBandManagementEnabledByZoneClusterHost(host);
+ if (Strings.isNullOrEmpty(newPassword)) {
+ throw new CloudRuntimeException(String.format("Cannot change out-of-band management password as provided new-password is null or empty for the host %s.", host.getUuid()));
+ }
+ GlobalLock outOfBandManagementHostLock = GlobalLock.getInternLock(getOutOfBandManagementHostLock(host.getId()));
+ try {
+ if (outOfBandManagementHostLock.lock(ACQUIRE_GLOBAL_LOCK_TIMEOUT_FOR_HOST)) {
--- End diff --
The nesting in this method is very deep and hard to follow. Please change this line to the following:
```
if (!outOfBandManagementHostLock.lock(ACQUIRE_GLOBAL_LOCK_TIMEOUT_FOR_HOST)) {
LOG.error("Unable to acquire synchronization lock to change out-of-band management password for host id: " + host.getId());
throw new CloudRuntimeException(String.format("Unable to acquire lock to change out-of-band management password for host (%s), please try after some time.", host.getUuid()));
}
```
This will eliminate the need for the ``else`` block and allow subsequent code to remove from the scope of the ``if`` block.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61781371
--- Diff: engine/schema/src/org/apache/cloudstack/outofbandmanagement/dao/OutOfBandManagementDaoImpl.java ---
@@ -0,0 +1,163 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.outofbandmanagement.dao;
+
+import com.cloud.utils.DateUtil;
+import com.cloud.utils.db.Attribute;
+import com.cloud.utils.db.DB;
+import com.cloud.utils.db.Filter;
+import com.cloud.utils.db.GenericDaoBase;
+import com.cloud.utils.db.SearchBuilder;
+import com.cloud.utils.db.SearchCriteria;
+import com.cloud.utils.db.TransactionLegacy;
+import com.cloud.utils.db.UpdateBuilder;
+import com.cloud.utils.exception.CloudRuntimeException;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagement;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagementVO;
+import org.apache.log4j.Logger;
+import org.springframework.stereotype.Component;
+
+import javax.ejb.Local;
+import java.sql.PreparedStatement;
+import java.sql.SQLException;
+import java.util.List;
+
+@DB
+@Component
+@Local(value = {OutOfBandManagementDao.class})
+public class OutOfBandManagementDaoImpl extends GenericDaoBase<OutOfBandManagementVO, Long> implements OutOfBandManagementDao {
+ private static final Logger LOG = Logger.getLogger(OutOfBandManagementDaoImpl.class);
+
+ private SearchBuilder<OutOfBandManagementVO> HostSearch;
+ private SearchBuilder<OutOfBandManagementVO> ManagementServerSearch;
+ private SearchBuilder<OutOfBandManagementVO> OutOfBandManagementOwnerSearch;
+ private SearchBuilder<OutOfBandManagementVO> StateUpdateSearch;
+
+ private Attribute PowerStateAttr;
+ private Attribute MsIdAttr;
+ private Attribute UpdateTimeAttr;
+
+ public OutOfBandManagementDaoImpl() {
+ super();
+
+ HostSearch = createSearchBuilder();
+ HostSearch.and("hostId", HostSearch.entity().getHostId(), SearchCriteria.Op.EQ);
+ HostSearch.done();
+
+ ManagementServerSearch = createSearchBuilder();
+ ManagementServerSearch.and("server", ManagementServerSearch.entity().getManagementServerId(), SearchCriteria.Op.EQ);
+ ManagementServerSearch.done();
+
+ OutOfBandManagementOwnerSearch = createSearchBuilder();
+ OutOfBandManagementOwnerSearch.and("server", OutOfBandManagementOwnerSearch.entity().getManagementServerId(), SearchCriteria.Op.EQ);
+ OutOfBandManagementOwnerSearch.or("serverNull", OutOfBandManagementOwnerSearch.entity().getManagementServerId(), SearchCriteria.Op.NULL);
+ OutOfBandManagementOwnerSearch.done();
+
+ StateUpdateSearch = createSearchBuilder();
+ StateUpdateSearch.and("status", StateUpdateSearch.entity().getPowerState(), SearchCriteria.Op.EQ);
+ StateUpdateSearch.and("id", StateUpdateSearch.entity().getId(), SearchCriteria.Op.EQ);
+ StateUpdateSearch.and("update", StateUpdateSearch.entity().getUpdateCount(), SearchCriteria.Op.EQ);
+ StateUpdateSearch.done();
+
+ PowerStateAttr = _allAttributes.get("powerState");
+ MsIdAttr = _allAttributes.get("managementServerId");
+ UpdateTimeAttr = _allAttributes.get("updateTime");
+ assert (PowerStateAttr != null && MsIdAttr != null && UpdateTimeAttr != null) : "Couldn't find one of these attributes";
+ }
+
+ @Override
+ public OutOfBandManagement findByHost(long hostId) {
+ SearchCriteria<OutOfBandManagementVO> sc = HostSearch.create("hostId", hostId);
+ return findOneBy(sc);
+ }
+
+ @Override
+ public List<OutOfBandManagementVO> findAllByManagementServer(long serverId) {
+ SearchCriteria<OutOfBandManagementVO> sc = OutOfBandManagementOwnerSearch.create();
+ sc.setParameters("server", serverId);
+ return listBy(sc, new Filter(OutOfBandManagementVO.class, "updateTime", true, null, null));
+ }
+
+ private void executeExpireOwnershipSql(final String sql, long resource) {
+ TransactionLegacy txn = TransactionLegacy.currentTxn();
+ try {
+ txn.start();
+ PreparedStatement pstmt = txn.prepareAutoCloseStatement(sql);
+ pstmt.setLong(1, resource);
+ pstmt.executeUpdate();
+ txn.commit();
+ } catch (SQLException e) {
+ txn.rollback();
+ throw new CloudRuntimeException("Unable to reset out-of-band management ownership based on resource:" + resource);
+ }
+ }
+
+ @Override
+ public void expireOutOfBandManagementOwnershipByHours(long hours) {
+ final String resetOwnerSql = "UPDATE oobm set mgmt_server_id=NULL where update_time<= (NOW() - INTERVAL ? HOUR)";
+ executeExpireOwnershipSql(resetOwnerSql, hours);
+ }
+
+ @Override
+ public void expireOutOfBandManagementOwnershipByServer(long serverId) {
+ final String resetOwnerSql = "UPDATE oobm set mgmt_server_id=NULL, power_state=NULL where mgmt_server_id=?";
+ executeExpireOwnershipSql(resetOwnerSql, serverId);
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Expired out-of-band management ownership for hosts owned by management server id:" + serverId);
+ }
+ }
+
+ @Override
+ public boolean updateState(OutOfBandManagement.PowerState oldStatus, OutOfBandManagement.PowerState.Event event, OutOfBandManagement.PowerState newStatus, OutOfBandManagement vo, Object data) {
+ // lock target row from beginning to avoid lock-promotion caused deadlock
+ OutOfBandManagementVO oobmHost = lockRow(vo.getId(), true);
+ if (oobmHost == null) {
+ if (LOG.isTraceEnabled()) {
+ LOG.trace("Failed to lock row to update state for out-of-band management host id: " + vo.getHostId());
+ }
+ return false;
+ }
+
+ Long newManagementServerId = event.getServerId();
+ // Avoid updates when old ownership and state are same as new
+ if (oldStatus == newStatus && (oobmHost.getManagementServerId() != null && oobmHost.getManagementServerId().equals(newManagementServerId))) {
+ return false;
+ }
+
+ if (event == OutOfBandManagement.PowerState.Event.Disabled) {
+ newManagementServerId = null;
+ }
+
+ SearchCriteria<OutOfBandManagementVO> sc = StateUpdateSearch.create();
+ sc.setParameters("status", oldStatus);
+ sc.setParameters("id", oobmHost.getId());
+ sc.setParameters("update", oobmHost.getUpdateCount());
+
+ oobmHost.incrUpdateCount();
--- End diff --
This method appears to be using both pessimistic locking (line #128) and optimistic locking here. Why lock the row if there is an optimistic lock as well.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61843036
--- Diff: api/src/org/apache/cloudstack/api/command/admin/outofbandmanagement/EnableOutOfBandManagementForZoneCmd.java ---
@@ -0,0 +1,105 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.api.command.admin.outofbandmanagement;
+
+import com.cloud.dc.DataCenter;
+import com.cloud.event.EventTypes;
+import com.cloud.exception.ConcurrentOperationException;
+import com.cloud.exception.InsufficientCapacityException;
+import com.cloud.exception.NetworkRuleConflictException;
+import com.cloud.exception.ResourceAllocationException;
+import com.cloud.exception.ResourceUnavailableException;
+import org.apache.cloudstack.acl.RoleType;
+import org.apache.cloudstack.api.APICommand;
+import org.apache.cloudstack.api.ApiConstants;
+import org.apache.cloudstack.api.ApiErrorCode;
+import org.apache.cloudstack.api.BaseAsyncCmd;
+import org.apache.cloudstack.api.BaseCmd;
+import org.apache.cloudstack.api.Parameter;
+import org.apache.cloudstack.api.ServerApiException;
+import org.apache.cloudstack.api.response.OutOfBandManagementResponse;
+import org.apache.cloudstack.api.response.ZoneResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagementService;
+
+import javax.inject.Inject;
+
+@APICommand(name = "enableOutOfBandManagementForZone", description = "Enables out-of-band management for a zone",
--- End diff --
fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-217944726
@rhtyd it looks like one of the OOBM smoke tests failed on the Travis build. Could you please investigate?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd closed the pull request at:
https://github.com/apache/cloudstack/pull/1502
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r62325927
--- Diff: utils/src/main/java/org/apache/cloudstack/utils/process/ProcessRunner.java ---
@@ -0,0 +1,111 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+
+package org.apache.cloudstack.utils.process;
+
+import org.apache.log4j.Logger;
+
+import java.io.IOException;
+import java.util.List;
+
+public class ProcessRunner {
+ public static final Logger LOG = Logger.getLogger(ProcessRunner.class);
+
+ private String stdOutput;
+ private String stdError;
+ private int returnCode = -1;
+
+ public String getStdOutput() {
+ return stdOutput;
+ }
+
+ public void setStdOutput(String stdOutput) {
+ this.stdOutput = stdOutput;
+ }
+
+ public String getStdError() {
+ return stdError;
+ }
+
+ public void setStdError(String stdError) {
+ this.stdError = stdError;
+ }
+
+ public int getReturnCode() {
+ return returnCode;
+ }
+
+ public void setReturnCode(int returnCode) {
+ this.returnCode = returnCode;
+ }
+
+ public static ProcessRunner executeCommands(List<String> commands, long timeOutSeconds) {
+ ProcessRunner result = new ProcessRunner();
+ try {
+ Process process = new ProcessBuilder().command(commands).start();
+ StreamGobbler stdInputGobbler = new StreamGobbler(process.getInputStream());
+ StreamGobbler stdErrorGobbler = new StreamGobbler(process.getErrorStream());
--- End diff --
The ``StreamGobbler`` idiom was replaced with ``ProcessBuilder.inheritIO`` in Java 1.7+. Using it would simplify this code and allow the removal of the ``StreamGobbler`` class.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-218945625
@swill I've tried to fix them here: https://github.com/apache/cloudstack/pull/1544
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-217318995
@swill I've added a Travis specific change and Travis specific patched ipmitool version (using patch from https://bugzilla.redhat.com/show_bug.cgi?id=1286035), I'm hoping with this the test should not cause random failure. If it happens to fail randomly in future, we can disable it from routine Travis runs.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-218421442
@nvazquez can you check why test_03_list_snapshots failed in the Travis run, with tearDown exception.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-217775304
@jburwell I've fixed the outstanding issues, please re-review and suggest any other improvement or LGTM. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r62325553
--- Diff: utils/src/main/java/org/apache/cloudstack/utils/process/ProcessRunner.java ---
@@ -0,0 +1,111 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+
+package org.apache.cloudstack.utils.process;
+
+import org.apache.log4j.Logger;
+
+import java.io.IOException;
+import java.util.List;
+
+public class ProcessRunner {
+ public static final Logger LOG = Logger.getLogger(ProcessRunner.class);
+
+ private String stdOutput;
+ private String stdError;
+ private int returnCode = -1;
+
+ public String getStdOutput() {
+ return stdOutput;
+ }
+
+ public void setStdOutput(String stdOutput) {
+ this.stdOutput = stdOutput;
+ }
+
+ public String getStdError() {
+ return stdError;
+ }
+
+ public void setStdError(String stdError) {
+ this.stdError = stdError;
+ }
+
+ public int getReturnCode() {
+ return returnCode;
+ }
+
+ public void setReturnCode(int returnCode) {
+ this.returnCode = returnCode;
+ }
+
+ public static ProcessRunner executeCommands(List<String> commands, long timeOutSeconds) {
+ ProcessRunner result = new ProcessRunner();
+ try {
+ Process process = new ProcessBuilder().command(commands).start();
+ StreamGobbler stdInputGobbler = new StreamGobbler(process.getInputStream());
+ StreamGobbler stdErrorGobbler = new StreamGobbler(process.getErrorStream());
+ stdInputGobbler.start();
+ stdErrorGobbler.start();
+
+ if (timeOutSeconds > 0) {
+ ProcessWaitForThread worker = new ProcessWaitForThread(process);
--- End diff --
I am concerned about unbounded thread growth here. Please consider using a bounded ``Executor`` and a [``ListenableFuture``](https://github.com/google/guava/wiki/ListenableFutureExplained) here. It would also remove the need for the ``ProcessWaitForThread`` class.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r62321148
--- Diff: api/src/org/apache/cloudstack/api/command/admin/outofbandmanagement/IssueOutOfBandManagementPowerActionCmd.java ---
@@ -0,0 +1,128 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.api.command.admin.outofbandmanagement;
+
+import com.cloud.event.EventTypes;
+import com.cloud.exception.ConcurrentOperationException;
+import com.cloud.exception.InsufficientCapacityException;
+import com.cloud.exception.NetworkRuleConflictException;
+import com.cloud.exception.ResourceAllocationException;
+import com.cloud.exception.ResourceUnavailableException;
+import com.cloud.host.Host;
+import com.google.common.base.Strings;
+import org.apache.cloudstack.acl.RoleType;
+import org.apache.cloudstack.api.APICommand;
+import org.apache.cloudstack.api.ApiCommandJobType;
+import org.apache.cloudstack.api.ApiConstants;
+import org.apache.cloudstack.api.ApiErrorCode;
+import org.apache.cloudstack.api.BaseAsyncCmd;
+import org.apache.cloudstack.api.Parameter;
+import org.apache.cloudstack.api.ServerApiException;
+import org.apache.cloudstack.api.response.HostResponse;
+import org.apache.cloudstack.api.response.OutOfBandManagementResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagement.PowerOperation;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagementService;
+
+import javax.inject.Inject;
+
+@APICommand(name = "issueOutOfBandManagementPowerAction", description = "Initiates the specified power action to the host's out-of-band management interface",
+ responseObject = OutOfBandManagementResponse.class, requestHasSensitiveInfo = false, responseHasSensitiveInfo = false, authorized = {RoleType.Admin})
+public class IssueOutOfBandManagementPowerActionCmd extends BaseAsyncCmd {
+ @Inject
+ private OutOfBandManagementService outOfBandManagementService;
+
+ /////////////////////////////////////////////////////
+ //////////////// API parameters /////////////////////
+ /////////////////////////////////////////////////////
+
+ @Parameter(name = ApiConstants.HOST_ID, type = CommandType.UUID, entityType = HostResponse.class, required = true, description = "the ID of the host")
+ private Long hostId;
+
+ @Parameter(name = ApiConstants.TIMEOUT, type = CommandType.LONG, description = "optional operation timeout in seconds that overrides the global or cluster-level out-of-band management timeout setting")
+ private Long actionTimeout;
+
+ @Parameter(name = ApiConstants.ACTION, type = CommandType.STRING, required = true, description = "out-of-band management power actions, valid actions are: ON, OFF, CYCLE, RESET, SOFT, STATUS")
+ private String outOfBandManagementPowerOperation;
+
+ /////////////////////////////////////////////////////
+ /////////////// API Implementation///////////////////
+ /////////////////////////////////////////////////////
+
+ @Override
+ public String getCommandName() {
+ return "issueoutofbandmanagementpoweractionresponse";
+ }
+
+ private void validateParams() {
+ if (getHostId() == null || getHostId() < 1L) {
+ throw new ServerApiException(ApiErrorCode.PARAM_ERROR, "Invalid host ID: " + getHostId());
+ }
+ if (Strings.isNullOrEmpty(getOutOfBandManagementPowerOperation())) {
+ throw new ServerApiException(ApiErrorCode.PARAM_ERROR, "Invalid out-of-band management power action: " + getOutOfBandManagementPowerOperation());
+ }
+ }
+
+ @Override
+ public void execute() throws ResourceUnavailableException, InsufficientCapacityException, ServerApiException, ConcurrentOperationException, ResourceAllocationException, NetworkRuleConflictException {
+ validateParams();
+ Host host = _resourceService.getHost(getHostId());
+ if (host == null) {
+ throw new ServerApiException(ApiErrorCode.PARAM_ERROR, "Unable to find host by ID: " + getHostId());
+ }
+ PowerOperation powerOperation = PowerOperation.valueOf(getOutOfBandManagementPowerOperation());
--- End diff --
@rhtyd the API itself is case-insensitive. Therefore, I am more concerned that the user will received an unexpected error because we are not fulfilling the contract. For this reason, it seems like we should be doing a case intensive check here and in ``validateParams``.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r62655993
--- Diff: utils/src/main/java/org/apache/cloudstack/utils/process/ProcessRunner.java ---
@@ -0,0 +1,109 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+
+package org.apache.cloudstack.utils.process;
+
+import com.cloud.utils.concurrency.NamedThreadFactory;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Strings;
+import org.apache.log4j.Logger;
+import org.joda.time.Duration;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.List;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+public class ProcessRunner {
+ public static final Logger LOG = Logger.getLogger(ProcessRunner.class);
+
+ private static final ExecutorService processExecutor = Executors.newCachedThreadPool(new NamedThreadFactory("ProcessRunner"));
+
+ private static String readStream(final InputStream inputStream) throws IOException {
+ final StringBuilder sb = new StringBuilder();
+ final BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(inputStream));
+ String line;
+ while ((line = bufferedReader.readLine()) != null) {
+ sb.append(line);
+ sb.append("\n");
+ }
+ return sb.toString();
+ }
+
+ public static ProcessResult executeCommands(final List<String> commands, final Duration timeOut) {
+ Preconditions.checkArgument(commands != null && timeOut != null);
+
+ int retVal = -2;
+ String stdOutput = null;
+ String stdError = null;
+
+ try {
+ final Process process = new ProcessBuilder().command(commands).start();
+ if (timeOut.getStandardSeconds() > 0) {
+ final Future<Integer> processFuture = processExecutor.submit(new Callable<Integer>() {
+ @Override
+ public Integer call() throws Exception {
+ return process.waitFor();
+ }
+ });
+ try {
+ retVal = processFuture.get(timeOut.getStandardSeconds(), TimeUnit.SECONDS);
+ } catch (ExecutionException e) {
+ retVal = -1;
+ stdError = e.getMessage();
+ if (LOG.isTraceEnabled()) {
+ LOG.trace("Failed to complete the requested command due to execution error: " + e.getMessage());
+ }
+ } catch (TimeoutException e) {
+ retVal = -1;
+ stdError = "Operation timed out, aborted";
+ if (LOG.isTraceEnabled()) {
+ LOG.trace("Failed to complete the requested command within timeout: " + e.getMessage());
+ }
+ } finally {
+ if (Strings.isNullOrEmpty(stdError)) {
+ stdOutput = readStream(process.getInputStream());
--- End diff --
Have you considered using Guava's ``CharStreams.toString(new BufferedWriter(process.getInputStream))`` instead of ``readStream``?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by swill <gi...@git.apache.org>.
Github user swill commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-214826838
@pdion891 sorry I kind of just threw you into this one. In this case there is a new requirement added for tests integration tests to pass, so it is not a blocking change, but I do think it is worth noting somewhere that new requirements were added for integration tests to pass. More generally, I have not seen any mechanism with which we track environment change requirements introduced in PRs that are then presented to the user in some way. I am assuming it makes sense that this data be in the release notes, but we need a way to flag the changes so they are easily discovered by the person who is generating the release notes (you in this case). :) Suggestions?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61813894
--- Diff: server/test/com/cloud/resource/MockResourceManagerImpl.java ---
@@ -172,6 +173,12 @@ public Cluster getCluster(final Long clusterId) {
return null;
}
+ @Override
+ public DataCenter getZone(Long zoneId) {
+ // TODO Auto-generated method stub
--- End diff --
Is there a TODO to be completed? If not, please remove the comment.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61787788
--- Diff: plugins/outofbandmanagement-drivers/ipmitool/src/org/apache/cloudstack/outofbandmanagement/driver/ipmitool/IpmitoolOutOfBandManagementDriver.java ---
@@ -0,0 +1,159 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.outofbandmanagement.driver.ipmitool;
+
+import com.cloud.utils.component.AdapterBase;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.google.common.base.Strings;
+import com.google.common.collect.ImmutableMap;
+import org.apache.cloudstack.framework.config.ConfigKey;
+import org.apache.cloudstack.framework.config.Configurable;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagement;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagementDriver;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverChangePasswordCommand;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverCommand;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverPowerCommand;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverResponse;
+import org.apache.log4j.Logger;
+
+import javax.ejb.Local;
+import java.util.Arrays;
+import java.util.List;
+
+@Local(value = {OutOfBandManagementDriver.class})
+public class IpmitoolOutOfBandManagementDriver extends AdapterBase implements OutOfBandManagementDriver, Configurable {
+ public static final Logger LOG = Logger.getLogger(IpmitoolOutOfBandManagementDriver.class);
+
+ private static volatile boolean isDriverEnabled = false;
+ private static boolean isIpmiToolBinAvailable = false;
--- End diff --
Why are the ``isDriverEnabled`` and ``isIpmiToolBinAvailable`` fields declared ``static``? They only appear to accessed from an instance context. Furthermore, since the path to IpmiTool cannot be changed unless the management server is restarted, the ``isIpmiToolBinAvailable`` field could be declared as a ``final`` instance variable with its value calculated in the default constructor.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r62612035
--- Diff: utils/src/main/java/org/apache/cloudstack/utils/process/ProcessRunner.java ---
@@ -0,0 +1,112 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+
+package org.apache.cloudstack.utils.process;
+
+import com.cloud.utils.concurrency.NamedThreadFactory;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Strings;
+import org.apache.log4j.Logger;
+import org.joda.time.Duration;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.List;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+public class ProcessRunner {
+ public static final Logger LOG = Logger.getLogger(ProcessRunner.class);
+
+ private static final ExecutorService processExecutor = Executors.newCachedThreadPool(new NamedThreadFactory("ProcessRunner"));
+
+ private static String readStream(final InputStream inputStream) {
+ String text = null;
+ try {
+ final BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(inputStream));
+ String line;
+ while ((line = bufferedReader.readLine()) != null) {
+ if (Strings.isNullOrEmpty(text)) {
+ text = line;
+ } else {
+ text = text + "\n" + line;
+ }
+ }
+ } catch (IOException e) {
+ if (LOG.isTraceEnabled()) {
+ LOG.trace("ProcessRunner::readStream failed due to: " + e.getMessage());
+ }
+ }
+ return text;
+ }
+
+ public static ProcessResult executeCommands(final List<String> commands, final Duration timeOut) {
+ Preconditions.checkArgument(commands != null && timeOut != null);
+
+ int retVal = -2;
+ String stdOutput = null;
+ String stdError = null;
+
+ try {
+ final Process process = new ProcessBuilder().command(commands).start();
+ if (timeOut.getStandardSeconds() > 0) {
--- End diff --
Yes, in case of initialization where we are checking the version etc a timeout may not be specified (0 = wait until process returns/exits). Also, this is a general utility not specifically tied to oobm.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61843007
--- Diff: api/src/org/apache/cloudstack/api/command/admin/outofbandmanagement/DisableOutOfBandManagementForHostCmd.java ---
@@ -0,0 +1,107 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.api.command.admin.outofbandmanagement;
+
+import com.cloud.event.EventTypes;
+import com.cloud.exception.ConcurrentOperationException;
+import com.cloud.exception.InsufficientCapacityException;
+import com.cloud.exception.NetworkRuleConflictException;
+import com.cloud.exception.ResourceAllocationException;
+import com.cloud.exception.ResourceUnavailableException;
+import com.cloud.host.Host;
+import org.apache.cloudstack.acl.RoleType;
+import org.apache.cloudstack.api.APICommand;
+import org.apache.cloudstack.api.ApiConstants;
+import org.apache.cloudstack.api.ApiErrorCode;
+import org.apache.cloudstack.api.BaseAsyncCmd;
+import org.apache.cloudstack.api.BaseCmd;
+import org.apache.cloudstack.api.Parameter;
+import org.apache.cloudstack.api.ServerApiException;
+import org.apache.cloudstack.api.response.HostResponse;
+import org.apache.cloudstack.api.response.OutOfBandManagementResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagementService;
+
+import javax.inject.Inject;
+
+@APICommand(name = "disableOutOfBandManagementForHost", description = "Disables out-of-band management for a host",
--- End diff --
fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61778682
--- Diff: api/src/org/apache/cloudstack/outofbandmanagement/OutOfBandManagementService.java ---
@@ -0,0 +1,51 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.outofbandmanagement;
+
+import com.cloud.dc.DataCenter;
+import com.cloud.host.Host;
+import com.cloud.org.Cluster;
+import com.google.common.collect.ImmutableMap;
+import org.apache.cloudstack.api.response.OutOfBandManagementResponse;
+import org.apache.cloudstack.framework.config.ConfigKey;
+
+import java.util.List;
+
+public interface OutOfBandManagementService {
+
+ ConfigKey<Long> OutOfBandManagementActionTimeout = new ConfigKey<Long>("Advanced", Long.class, "outofbandmanagement.action.timeout", "60",
+ "The out of band management action timeout in seconds, configurable by cluster", true, ConfigKey.Scope.Cluster);
+
+ ConfigKey<Long> OutOfBandManagementSyncThreadInterval = new ConfigKey<Long>("Advanced", Long.class, "outofbandmanagement.sync.interval", "300000",
+ "The interval (in milliseconds) when the out-of-band management background sync are retrieved", true, ConfigKey.Scope.Global);
+
+ ConfigKey<Integer> OutOfBandManagementSyncThreadPoolSize = new ConfigKey<Integer>("Advanced", Integer.class, "outofbandmanagement.sync.poolsize", "50",
+ "The out of band management background sync thread pool size", true, ConfigKey.Scope.Global);
+
+ long getId();
+ boolean isOutOfBandManagementEnabled(Host host);
+ void submitBackgroundPowerSyncTask(Host host);
+ boolean transitionPowerStateToDisabled(List<? extends Host> hosts);
+
+ OutOfBandManagementResponse enableDisableOutOfBandManagement(DataCenter zone, boolean enabled);
+ OutOfBandManagementResponse enableDisableOutOfBandManagement(Cluster cluster, boolean enabled);
+ OutOfBandManagementResponse enableDisableOutOfBandManagement(Host host, boolean enabled);
--- End diff --
Why not name these methods either ``enableOutOfBandManagement`` or ``disableOutOfBandManagement``? The naming seems ambiguous as it does not indicate what a ``true`` or ``false`` value means for the ``enabled`` parameter.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by swill <gi...@git.apache.org>.
Github user swill commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-216274536
I think this one is ready to merge...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by pdion891 <gi...@git.apache.org>.
Github user pdion891 commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-214840213
shouldn't we have a wiki page explaining how to do tests agains cloudstack? and this would be part of this page or a subpage ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-214887358
@swill I am going to start in my review later this evening. Sorry for the delay.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by bhaisaab <gi...@git.apache.org>.
Github user bhaisaab commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r60727764
--- Diff: api/src/org/apache/cloudstack/api/response/ZoneResponse.java ---
@@ -233,6 +234,9 @@ public void addTag(ResourceTagResponse tag) {
}
public void setResourceDetails(Map<String, String> details) {
- this.resourceDetails = details;
+ if (details == null) {
--- End diff --
yes, if its null no point in setting it as nothing will be marshaled by gson/xml processor, also we don't want to consume the map but create a new copy out of it to avoid polluting user's datastructures
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: [WIP] Don't start review yet -- CLOUDSTAC...
Posted by bhaisaab <gi...@git.apache.org>.
Github user bhaisaab commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-212340052
UI Screenshots;
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-214858085
@swill @pdion891 thanks, I don't know of any format or wiki/document where we keep this thing. Each feature/marvin-test may require test specific environment so I'm not sure if we can have a general solution.
In general, for writing marvin tests new dependencies maybe added to marvin itself (like for this PR, `ipmisim` dependency is added to marvin, so anytime you install it you automatically get the dependency necessary by your Python based test). Along with this, you may need any external dependencies added to packaging requirements/dependency (for example, `ipmitool` added to debian control, while for rpms `ipmitool` was already in the list of dependencies for cloudstack-management package). For running tests, we should expect the test to work out of the box provided the dependencies are installed. For this PR, I've got `ipmisim` added to marvin's setup.py, and for Travis specific there is a apt-get install ipmitool. After this the oobm marvin test runs like any other test and internally is responsible for setting up (fake) ipmi server etc.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by swill <gi...@git.apache.org>.
Github user swill commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-213605569
Perfect thanks @DaanHoogland. Once we get more people using bubble, maybe we can start posting to the PR that we are kicking off a CI run (like you did here) so we can be sure to best use our limited resources. I have been helping @kiwiflyer and @dmabry get bubble setup in their environment and I think they are all set now, so they will likely be contributing to this effort as well.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by wido <gi...@git.apache.org>.
Github user wido commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-213702911
I added a few comments, but before I continue:
- Why assume a IP-Address and not a hostname OR IP? I know enough IPMIs why are available through a DNS hostname
- Why is so much passed as a String through the code instead of a native type like integer or inetaddress
My first look looks good, but I can't test this easily.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r62616710
--- Diff: utils/src/main/java/org/apache/cloudstack/utils/process/ProcessRunner.java ---
@@ -0,0 +1,112 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+
+package org.apache.cloudstack.utils.process;
+
+import com.cloud.utils.concurrency.NamedThreadFactory;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Strings;
+import org.apache.log4j.Logger;
+import org.joda.time.Duration;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.List;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+public class ProcessRunner {
+ public static final Logger LOG = Logger.getLogger(ProcessRunner.class);
+
+ private static final ExecutorService processExecutor = Executors.newCachedThreadPool(new NamedThreadFactory("ProcessRunner"));
+
+ private static String readStream(final InputStream inputStream) {
+ String text = null;
+ try {
+ final BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(inputStream));
+ String line;
+ while ((line = bufferedReader.readLine()) != null) {
+ if (Strings.isNullOrEmpty(text)) {
+ text = line;
+ } else {
+ text = text + "\n" + line;
+ }
+ }
+ } catch (IOException e) {
+ if (LOG.isTraceEnabled()) {
+ LOG.trace("ProcessRunner::readStream failed due to: " + e.getMessage());
+ }
+ }
+ return text;
+ }
+
+ public static ProcessResult executeCommands(final List<String> commands, final Duration timeOut) {
+ Preconditions.checkArgument(commands != null && timeOut != null);
+
+ int retVal = -2;
+ String stdOutput = null;
+ String stdError = null;
+
+ try {
+ final Process process = new ProcessBuilder().command(commands).start();
+ if (timeOut.getStandardSeconds() > 0) {
+ final Future<Integer> processFuture = processExecutor.submit(new Callable<Integer>() {
+ @Override
+ public Integer call() throws Exception {
+ return process.waitFor();
+ }
+ });
+ try {
+ retVal = processFuture.get(timeOut.getStandardSeconds(), TimeUnit.SECONDS);
+ } catch (ExecutionException | TimeoutException e) {
--- End diff --
Fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r62441896
--- Diff: server/src/com/cloud/server/StatsCollector.java ---
@@ -412,6 +428,36 @@ protected void runInContext() {
}
}
+ class HostOutOfBandManagementStatsCollector extends ManagedContextRunnable {
+ @Override
+ protected void runInContext() {
+ try {
+ s_logger.debug("HostOutOfBandManagementStatsCollector is running...");
+ List<OutOfBandManagementVO> outOfBandManagementHosts = outOfBandManagementDao.findAllByManagementServer(ManagementServerNode.getManagementServerId());
+ if (outOfBandManagementHosts == null) {
+ return;
+ }
+ for (OutOfBandManagement outOfBandManagementHost : outOfBandManagementHosts) {
+ Host host = _hostDao.findById(outOfBandManagementHost.getHostId());
+ if (host == null) {
+ continue;
+ }
+ if (outOfBandManagementService.isOutOfBandManagementEnabled(host)) {
+ outOfBandManagementService.submitBackgroundPowerSyncTask(host);
+ } else {
+ if (outOfBandManagementHost.getPowerState() != OutOfBandManagement.PowerState.Disabled) {
+ if (outOfBandManagementService.transitionPowerStateToDisabled(Collections.singletonList(host))) {
+ s_logger.debug("Out-of-band management was disabled in zone/cluster/host, disabled power state for host id:" + host.getId());
--- End diff --
Fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61781601
--- Diff: engine/schema/src/org/apache/cloudstack/outofbandmanagement/dao/OutOfBandManagementDaoImpl.java ---
@@ -0,0 +1,163 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.outofbandmanagement.dao;
+
+import com.cloud.utils.DateUtil;
+import com.cloud.utils.db.Attribute;
+import com.cloud.utils.db.DB;
+import com.cloud.utils.db.Filter;
+import com.cloud.utils.db.GenericDaoBase;
+import com.cloud.utils.db.SearchBuilder;
+import com.cloud.utils.db.SearchCriteria;
+import com.cloud.utils.db.TransactionLegacy;
+import com.cloud.utils.db.UpdateBuilder;
+import com.cloud.utils.exception.CloudRuntimeException;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagement;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagementVO;
+import org.apache.log4j.Logger;
+import org.springframework.stereotype.Component;
+
+import javax.ejb.Local;
+import java.sql.PreparedStatement;
+import java.sql.SQLException;
+import java.util.List;
+
+@DB
+@Component
+@Local(value = {OutOfBandManagementDao.class})
+public class OutOfBandManagementDaoImpl extends GenericDaoBase<OutOfBandManagementVO, Long> implements OutOfBandManagementDao {
+ private static final Logger LOG = Logger.getLogger(OutOfBandManagementDaoImpl.class);
+
+ private SearchBuilder<OutOfBandManagementVO> HostSearch;
+ private SearchBuilder<OutOfBandManagementVO> ManagementServerSearch;
+ private SearchBuilder<OutOfBandManagementVO> OutOfBandManagementOwnerSearch;
+ private SearchBuilder<OutOfBandManagementVO> StateUpdateSearch;
+
+ private Attribute PowerStateAttr;
+ private Attribute MsIdAttr;
+ private Attribute UpdateTimeAttr;
+
+ public OutOfBandManagementDaoImpl() {
+ super();
+
+ HostSearch = createSearchBuilder();
+ HostSearch.and("hostId", HostSearch.entity().getHostId(), SearchCriteria.Op.EQ);
+ HostSearch.done();
+
+ ManagementServerSearch = createSearchBuilder();
+ ManagementServerSearch.and("server", ManagementServerSearch.entity().getManagementServerId(), SearchCriteria.Op.EQ);
+ ManagementServerSearch.done();
+
+ OutOfBandManagementOwnerSearch = createSearchBuilder();
+ OutOfBandManagementOwnerSearch.and("server", OutOfBandManagementOwnerSearch.entity().getManagementServerId(), SearchCriteria.Op.EQ);
+ OutOfBandManagementOwnerSearch.or("serverNull", OutOfBandManagementOwnerSearch.entity().getManagementServerId(), SearchCriteria.Op.NULL);
+ OutOfBandManagementOwnerSearch.done();
+
+ StateUpdateSearch = createSearchBuilder();
+ StateUpdateSearch.and("status", StateUpdateSearch.entity().getPowerState(), SearchCriteria.Op.EQ);
+ StateUpdateSearch.and("id", StateUpdateSearch.entity().getId(), SearchCriteria.Op.EQ);
+ StateUpdateSearch.and("update", StateUpdateSearch.entity().getUpdateCount(), SearchCriteria.Op.EQ);
+ StateUpdateSearch.done();
+
+ PowerStateAttr = _allAttributes.get("powerState");
+ MsIdAttr = _allAttributes.get("managementServerId");
+ UpdateTimeAttr = _allAttributes.get("updateTime");
+ assert (PowerStateAttr != null && MsIdAttr != null && UpdateTimeAttr != null) : "Couldn't find one of these attributes";
+ }
+
+ @Override
+ public OutOfBandManagement findByHost(long hostId) {
+ SearchCriteria<OutOfBandManagementVO> sc = HostSearch.create("hostId", hostId);
+ return findOneBy(sc);
+ }
+
+ @Override
+ public List<OutOfBandManagementVO> findAllByManagementServer(long serverId) {
+ SearchCriteria<OutOfBandManagementVO> sc = OutOfBandManagementOwnerSearch.create();
+ sc.setParameters("server", serverId);
+ return listBy(sc, new Filter(OutOfBandManagementVO.class, "updateTime", true, null, null));
+ }
+
+ private void executeExpireOwnershipSql(final String sql, long resource) {
+ TransactionLegacy txn = TransactionLegacy.currentTxn();
+ try {
+ txn.start();
+ PreparedStatement pstmt = txn.prepareAutoCloseStatement(sql);
+ pstmt.setLong(1, resource);
+ pstmt.executeUpdate();
+ txn.commit();
+ } catch (SQLException e) {
+ txn.rollback();
+ throw new CloudRuntimeException("Unable to reset out-of-band management ownership based on resource:" + resource);
+ }
+ }
+
+ @Override
+ public void expireOutOfBandManagementOwnershipByHours(long hours) {
+ final String resetOwnerSql = "UPDATE oobm set mgmt_server_id=NULL where update_time<= (NOW() - INTERVAL ? HOUR)";
+ executeExpireOwnershipSql(resetOwnerSql, hours);
+ }
+
+ @Override
+ public void expireOutOfBandManagementOwnershipByServer(long serverId) {
+ final String resetOwnerSql = "UPDATE oobm set mgmt_server_id=NULL, power_state=NULL where mgmt_server_id=?";
+ executeExpireOwnershipSql(resetOwnerSql, serverId);
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Expired out-of-band management ownership for hosts owned by management server id:" + serverId);
+ }
+ }
+
+ @Override
+ public boolean updateState(OutOfBandManagement.PowerState oldStatus, OutOfBandManagement.PowerState.Event event, OutOfBandManagement.PowerState newStatus, OutOfBandManagement vo, Object data) {
+ // lock target row from beginning to avoid lock-promotion caused deadlock
+ OutOfBandManagementVO oobmHost = lockRow(vo.getId(), true);
+ if (oobmHost == null) {
+ if (LOG.isTraceEnabled()) {
+ LOG.trace("Failed to lock row to update state for out-of-band management host id: " + vo.getHostId());
+ }
+ return false;
+ }
+
+ Long newManagementServerId = event.getServerId();
+ // Avoid updates when old ownership and state are same as new
+ if (oldStatus == newStatus && (oobmHost.getManagementServerId() != null && oobmHost.getManagementServerId().equals(newManagementServerId))) {
+ return false;
+ }
+
+ if (event == OutOfBandManagement.PowerState.Event.Disabled) {
+ newManagementServerId = null;
+ }
+
+ SearchCriteria<OutOfBandManagementVO> sc = StateUpdateSearch.create();
+ sc.setParameters("status", oldStatus);
+ sc.setParameters("id", oobmHost.getId());
+ sc.setParameters("update", oobmHost.getUpdateCount());
+
+ oobmHost.incrUpdateCount();
+ UpdateBuilder ub = getUpdateBuilder(oobmHost);
+ ub.set(oobmHost, PowerStateAttr, newStatus);
+ ub.set(oobmHost, UpdateTimeAttr, DateUtil.currentGMTTime());
+ ub.set(oobmHost, MsIdAttr, newManagementServerId);
--- End diff --
The update counter does not appear to be updated in this statement then the value will not progress -- preventing the optimistic locking from working as expected.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61777183
--- Diff: api/src/org/apache/cloudstack/api/command/admin/outofbandmanagement/EnableOutOfBandManagementForHostCmd.java ---
@@ -0,0 +1,107 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.api.command.admin.outofbandmanagement;
+
+import com.cloud.event.EventTypes;
+import com.cloud.exception.ConcurrentOperationException;
+import com.cloud.exception.InsufficientCapacityException;
+import com.cloud.exception.NetworkRuleConflictException;
+import com.cloud.exception.ResourceAllocationException;
+import com.cloud.exception.ResourceUnavailableException;
+import com.cloud.host.Host;
+import org.apache.cloudstack.acl.RoleType;
+import org.apache.cloudstack.api.APICommand;
+import org.apache.cloudstack.api.ApiConstants;
+import org.apache.cloudstack.api.ApiErrorCode;
+import org.apache.cloudstack.api.BaseAsyncCmd;
+import org.apache.cloudstack.api.BaseCmd;
+import org.apache.cloudstack.api.Parameter;
+import org.apache.cloudstack.api.ServerApiException;
+import org.apache.cloudstack.api.response.HostResponse;
+import org.apache.cloudstack.api.response.OutOfBandManagementResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagementService;
+
+import javax.inject.Inject;
+
+@APICommand(name = "enableOutOfBandManagementForHost", description = "Enables out-of-band management for a host",
--- End diff --
Please add the version to this annotation
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r60890292
--- Diff: api/src/org/apache/cloudstack/api/command/admin/outofbandmanagement/ConfigureOutOfBandManagementCmd.java ---
@@ -0,0 +1,127 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.api.command.admin.outofbandmanagement;
+
+import com.cloud.exception.ConcurrentOperationException;
+import com.cloud.exception.InsufficientCapacityException;
+import com.cloud.exception.NetworkRuleConflictException;
+import com.cloud.exception.ResourceAllocationException;
+import com.cloud.exception.ResourceUnavailableException;
+import com.cloud.host.Host;
+import com.google.common.base.Strings;
+import com.google.common.collect.ImmutableMap;
+import org.apache.cloudstack.acl.RoleType;
+import org.apache.cloudstack.api.APICommand;
+import org.apache.cloudstack.api.ApiConstants;
+import org.apache.cloudstack.api.ApiErrorCode;
+import org.apache.cloudstack.api.BaseCmd;
+import org.apache.cloudstack.api.Parameter;
+import org.apache.cloudstack.api.ServerApiException;
+import org.apache.cloudstack.api.response.HostResponse;
+import org.apache.cloudstack.api.response.OutOfBandManagementResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagement;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagementService;
+
+import javax.inject.Inject;
+
+@APICommand(name = "configureOutOfBandManagement", description = "Configures a host's out-of-band management interface",
+ responseObject = OutOfBandManagementResponse.class, requestHasSensitiveInfo = true, responseHasSensitiveInfo = false, authorized = {RoleType.Admin})
+public class ConfigureOutOfBandManagementCmd extends BaseCmd {
+ @Inject
+ private OutOfBandManagementService outOfBandManagementService;
+
+ /////////////////////////////////////////////////////
+ //////////////// API parameters /////////////////////
+ /////////////////////////////////////////////////////
+
+ @Parameter(name = ApiConstants.HOST_ID, type = CommandType.UUID, entityType = HostResponse.class, required = true, description = "the ID of the host")
+ private Long hostId;
+
+ @Parameter(name = ApiConstants.DRIVER, type = CommandType.STRING, required = true, description = "the host management interface driver, for example: ipmitool")
+ private String driver;
+
+ @Parameter(name = ApiConstants.IP_ADDRESS, type = CommandType.STRING, required = true, description = "the host management interface IP address")
--- End diff --
Fixed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61777066
--- Diff: api/src/org/apache/cloudstack/api/command/admin/outofbandmanagement/DisableOutOfBandManagementForZoneCmd.java ---
@@ -0,0 +1,105 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.api.command.admin.outofbandmanagement;
+
+import com.cloud.dc.DataCenter;
+import com.cloud.event.EventTypes;
+import com.cloud.exception.ConcurrentOperationException;
+import com.cloud.exception.InsufficientCapacityException;
+import com.cloud.exception.NetworkRuleConflictException;
+import com.cloud.exception.ResourceAllocationException;
+import com.cloud.exception.ResourceUnavailableException;
+import org.apache.cloudstack.acl.RoleType;
+import org.apache.cloudstack.api.APICommand;
+import org.apache.cloudstack.api.ApiConstants;
+import org.apache.cloudstack.api.ApiErrorCode;
+import org.apache.cloudstack.api.BaseAsyncCmd;
+import org.apache.cloudstack.api.BaseCmd;
+import org.apache.cloudstack.api.Parameter;
+import org.apache.cloudstack.api.ServerApiException;
+import org.apache.cloudstack.api.response.OutOfBandManagementResponse;
+import org.apache.cloudstack.api.response.ZoneResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagementService;
+
+import javax.inject.Inject;
+
+@APICommand(name = "disableOutOfBandManagementForZone", description = "Disables out-of-band management for a zone",
--- End diff --
Please add the version to this annotation
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61847657
--- Diff: test/integration/smoke/test_outofbandmanagement.py ---
@@ -0,0 +1,561 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import marvin
+from marvin.cloudstackTestCase import *
+from marvin.cloudstackAPI import *
+from marvin.lib.utils import *
+from marvin.lib.base import *
+from marvin.lib.common import *
+from marvin.lib.utils import (random_gen)
+from nose.plugins.attrib import attr
+
+from ipmisim.ipmisim import IpmiServerContext, IpmiServer, ThreadedIpmiServer
+
+import socket
+import sys
+import thread
+import time
+
+
+class TestOutOfBandManagement(cloudstackTestCase):
+ """ Test cases for out of band management
+ """
+
+ def setUp(self):
+ self.apiclient = self.testClient.getApiClient()
+ self.hypervisor = self.testClient.getHypervisorInfo()
+ self.dbclient = self.testClient.getDbConnection()
+ self.services = self.testClient.getParsedTestDataConfig()
+ self.mgtSvrDetails = self.config.__dict__["mgtSvr"][0].__dict__
+
+ self.zone = get_zone(self.apiclient, self.testClient.getZoneForTests())
+ self.host = None
+ self.server = None
+
+ # use random port for ipmisim
+ s = socket.socket()
+ s.bind(('', 0))
+ self.serverPort = s.getsockname()[1]
+ s.close()
+
+ self.cleanup = []
+
+
+ def tearDown(self):
+ try:
+ self.dbclient.execute("delete from oobm where port=%d" % self.getIpmiServerPort())
+ self.dbclient.execute("delete from mshost_peer where peer_runid=%s" % self.getFakeMsRunId())
+ self.dbclient.execute("delete from mshost where runid=%s" % self.getFakeMsRunId())
+ self.dbclient.execute("delete from cluster_details where name='outOfBandManagementEnabled'")
+ self.dbclient.execute("delete from data_center_details where name='outOfBandManagementEnabled'")
+ cleanup_resources(self.apiclient, self.cleanup)
+ if self.server:
+ self.server.shutdown()
+ self.server.server_close()
+ except Exception as e:
+ raise Exception("Warning: Exception during cleanup : %s" % e)
+
+
+ def getFakeMsId(self):
+ return 1234567890
+
+
+ def getFakeMsRunId(self):
+ return 123456
+
+
+ def getHost(self, hostId=None):
+ if self.host and hostId is None:
+ return self.host
+
+ response = list_hosts(
+ self.apiclient,
+ zoneid=self.zone.id,
+ type='Routing',
+ id=hostId
+ )
+ if len(response) > 0:
+ self.host = response[0]
+ return self.host
+ raise self.skipTest("No hosts found, skipping out-of-band management test")
+
+
+ def getIpmiServerIp(self):
+ s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
+ s.connect((self.mgtSvrDetails["mgtSvrIp"], self.mgtSvrDetails["port"]))
+ return s.getsockname()[0]
+
+
+ def getIpmiServerPort(self):
+ return self.serverPort
+
+
+ def getOobmConfigCmd(self):
+ cmd = configureOutOfBandManagement.configureOutOfBandManagementCmd()
+ cmd.driver = 'ipmitool' # The default available driver
+ cmd.address = self.getIpmiServerIp()
+ cmd.port = self.getIpmiServerPort()
+ cmd.username = 'admin'
+ cmd.password = 'password'
+ cmd.hostid = self.getHost().id
+ return cmd
+
+
+ def getOobmEnableCmd(self):
+ cmd = enableOutOfBandManagementForHost.enableOutOfBandManagementForHostCmd()
+ cmd.hostid = self.getHost().id
+ return cmd
+
+
+ def getOobmDisableCmd(self):
+ cmd = disableOutOfBandManagementForHost.disableOutOfBandManagementForHostCmd()
+ cmd.hostid = self.getHost().id
+ return cmd
+
+
+ def getOobmIssueActionCmd(self):
+ cmd = issueOutOfBandManagementPowerAction.issueOutOfBandManagementPowerActionCmd()
+ cmd.hostid = self.getHost().id
+ cmd.action = 'STATUS'
+ return cmd
+
+
+ def issuePowerActionCmd(self, action, timeout=None):
+ cmd = self.getOobmIssueActionCmd()
+ cmd.action = action
+ if timeout:
+ cmd.timeout = timeout
+ return self.apiclient.issueOutOfBandManagementPowerAction(cmd)
+
+
+ def configureAndEnableOobm(self):
+ self.apiclient.configureOutOfBandManagement(self.getOobmConfigCmd())
+ response = self.apiclient.enableOutOfBandManagementForHost(self.getOobmEnableCmd())
+ self.assertEqual(response.enabled, True)
+
+
+ def startIpmiServer(self):
+ def startIpmiServer(tname, server):
+ self.debug("Starting ipmisim server")
+ try:
+ server.serve_forever()
+ except Exception: pass
+ IpmiServerContext('reset')
+ ThreadedIpmiServer.allow_reuse_address = False
+ server = ThreadedIpmiServer(('0.0.0.0', self.getIpmiServerPort()), IpmiServer)
+ thread.start_new_thread(startIpmiServer, ("ipmi-server", server,))
+ self.server = server
+
+
+ def checkSyncToState(self, state, interval):
+ self.debug("Waiting for background thread to update powerstate to " + state)
+ time.sleep(1 + int(interval)*2/1000) # interval is in ms
+ response = self.getHost(hostId=self.getHost().id).outofbandmanagement
+ self.assertEqual(response.powerstate, state)
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_configure_invalid_driver(self):
+ """
+ Tests out-of-band management configuration with invalid driver
+ """
+ cmd = self.getOobmConfigCmd()
+ cmd.driver = 'randomDriverThatDoesNotExist'
+ try:
+ response = self.apiclient.configureOutOfBandManagement(cmd)
+ self.fail("Expected an exception to be thrown, failing")
+ except Exception: pass
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_configure_default_driver(self):
+ """
+ Tests out-of-band management configuration with valid data
+ """
+ cmd = self.getOobmConfigCmd()
+ response = self.apiclient.configureOutOfBandManagement(cmd)
+ self.assertEqual(response.hostid, cmd.hostid)
+ self.assertEqual(response.driver, cmd.driver)
+ self.assertEqual(response.address, cmd.address)
+ self.assertEqual(response.port, str(cmd.port))
+ self.assertEqual(response.username, cmd.username)
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_enable_feature_invalid(self):
+ """
+ Tests out-of-band management host enable feature with
+ invalid options
+ """
+ cmd = self.getOobmEnableCmd()
+ cmd.hostid = -1
+ try:
+ response = self.apiclient.enableOutOfBandManagementForHost(cmd)
+ self.fail("Expected an exception to be thrown, failing")
+ except Exception: pass
+
+ try:
+ cmd = enableOutOfBandManagementForCluster.enableOutOfBandManagementForClusterCmd()
+ response = self.apiclient.enableOutOfBandManagementForCluster(cmd)
+ self.fail("Expected an exception to be thrown, failing")
+ except Exception: pass
+
+ try:
+ cmd = enableOutOfBandManagementForZone.enableOutOfBandManagementForZoneCmd()
+ response = self.apiclient.enableOutOfBandManagementForZone(cmd)
+ self.fail("Expected an exception to be thrown, failing")
+ except Exception: pass
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_disable_feature_invalid(self):
+ """
+ Tests out-of-band management host disable feature with
+ invalid options
+ """
+ cmd = self.getOobmDisableCmd()
+ cmd.hostid = -1
+ try:
+ response = self.apiclient.disableOutOfBandManagementForHost(cmd)
+ self.fail("Expected an exception to be thrown, failing")
+ except Exception: pass
+
+ try:
+ cmd = disableOutOfBandManagementForCluster.disableOutOfBandManagementForClusterCmd()
+ response = self.apiclient.disableOutOfBandManagementForCluster(cmd)
+ self.fail("Expected an exception to be thrown, failing")
+ except Exception: pass
+
+ try:
+ cmd = disableOutOfBandManagementForZone.disableOutOfBandManagementForZoneCmd()
+ response = self.apiclient.disableOutOfBandManagementForZone(cmd)
+ self.fail("Expected an exception to be thrown, failing")
+ except Exception: pass
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_enable_feature_valid(self):
+ """
+ Tests out-of-band management host enable feature with
+ valid options
+ """
+ self.apiclient.configureOutOfBandManagement(self.getOobmConfigCmd())
+ cmd = self.getOobmEnableCmd()
+ response = self.apiclient.enableOutOfBandManagementForHost(cmd)
+ self.assertEqual(response.hostid, cmd.hostid)
+ self.assertEqual(response.enabled, True)
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_disable_feature_valid(self):
+ """
+ Tests out-of-band management host disable feature with
+ valid options
+ """
+
+ self.apiclient.configureOutOfBandManagement(self.getOobmConfigCmd())
+ cmd = self.getOobmDisableCmd()
+ response = self.apiclient.disableOutOfBandManagementForHost(cmd)
+ self.assertEqual(response.hostid, cmd.hostid)
+ self.assertEqual(response.enabled, False)
+
+ response = self.getHost(hostId=cmd.hostid).outofbandmanagement
+ self.assertEqual(response.powerstate, 'Disabled')
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_enabledisable_across_clusterzones(self):
--- End diff --
the test case is dependent on cluster/zone level enable/disable setting; therefore they are configured as such
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61816189
--- Diff: test/integration/smoke/test_outofbandmanagement.py ---
@@ -0,0 +1,561 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import marvin
+from marvin.cloudstackTestCase import *
+from marvin.cloudstackAPI import *
+from marvin.lib.utils import *
+from marvin.lib.base import *
+from marvin.lib.common import *
+from marvin.lib.utils import (random_gen)
+from nose.plugins.attrib import attr
+
+from ipmisim.ipmisim import IpmiServerContext, IpmiServer, ThreadedIpmiServer
+
+import socket
+import sys
+import thread
+import time
+
+
+class TestOutOfBandManagement(cloudstackTestCase):
+ """ Test cases for out of band management
+ """
+
+ def setUp(self):
+ self.apiclient = self.testClient.getApiClient()
+ self.hypervisor = self.testClient.getHypervisorInfo()
+ self.dbclient = self.testClient.getDbConnection()
+ self.services = self.testClient.getParsedTestDataConfig()
+ self.mgtSvrDetails = self.config.__dict__["mgtSvr"][0].__dict__
+
+ self.zone = get_zone(self.apiclient, self.testClient.getZoneForTests())
+ self.host = None
+ self.server = None
+
+ # use random port for ipmisim
+ s = socket.socket()
+ s.bind(('', 0))
+ self.serverPort = s.getsockname()[1]
+ s.close()
+
+ self.cleanup = []
+
+
+ def tearDown(self):
+ try:
+ self.dbclient.execute("delete from oobm where port=%d" % self.getIpmiServerPort())
+ self.dbclient.execute("delete from mshost_peer where peer_runid=%s" % self.getFakeMsRunId())
+ self.dbclient.execute("delete from mshost where runid=%s" % self.getFakeMsRunId())
+ self.dbclient.execute("delete from cluster_details where name='outOfBandManagementEnabled'")
+ self.dbclient.execute("delete from data_center_details where name='outOfBandManagementEnabled'")
+ cleanup_resources(self.apiclient, self.cleanup)
+ if self.server:
+ self.server.shutdown()
+ self.server.server_close()
+ except Exception as e:
+ raise Exception("Warning: Exception during cleanup : %s" % e)
+
+
+ def getFakeMsId(self):
+ return 1234567890
+
+
+ def getFakeMsRunId(self):
+ return 123456
+
+
+ def getHost(self, hostId=None):
+ if self.host and hostId is None:
+ return self.host
+
+ response = list_hosts(
+ self.apiclient,
+ zoneid=self.zone.id,
+ type='Routing',
+ id=hostId
+ )
+ if len(response) > 0:
+ self.host = response[0]
+ return self.host
+ raise self.skipTest("No hosts found, skipping out-of-band management test")
+
+
+ def getIpmiServerIp(self):
+ s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
+ s.connect((self.mgtSvrDetails["mgtSvrIp"], self.mgtSvrDetails["port"]))
+ return s.getsockname()[0]
+
+
+ def getIpmiServerPort(self):
+ return self.serverPort
+
+
+ def getOobmConfigCmd(self):
+ cmd = configureOutOfBandManagement.configureOutOfBandManagementCmd()
+ cmd.driver = 'ipmitool' # The default available driver
+ cmd.address = self.getIpmiServerIp()
+ cmd.port = self.getIpmiServerPort()
+ cmd.username = 'admin'
+ cmd.password = 'password'
+ cmd.hostid = self.getHost().id
+ return cmd
+
+
+ def getOobmEnableCmd(self):
+ cmd = enableOutOfBandManagementForHost.enableOutOfBandManagementForHostCmd()
+ cmd.hostid = self.getHost().id
+ return cmd
+
+
+ def getOobmDisableCmd(self):
+ cmd = disableOutOfBandManagementForHost.disableOutOfBandManagementForHostCmd()
+ cmd.hostid = self.getHost().id
+ return cmd
+
+
+ def getOobmIssueActionCmd(self):
+ cmd = issueOutOfBandManagementPowerAction.issueOutOfBandManagementPowerActionCmd()
+ cmd.hostid = self.getHost().id
+ cmd.action = 'STATUS'
+ return cmd
+
+
+ def issuePowerActionCmd(self, action, timeout=None):
+ cmd = self.getOobmIssueActionCmd()
+ cmd.action = action
+ if timeout:
+ cmd.timeout = timeout
+ return self.apiclient.issueOutOfBandManagementPowerAction(cmd)
+
+
+ def configureAndEnableOobm(self):
+ self.apiclient.configureOutOfBandManagement(self.getOobmConfigCmd())
+ response = self.apiclient.enableOutOfBandManagementForHost(self.getOobmEnableCmd())
+ self.assertEqual(response.enabled, True)
+
+
+ def startIpmiServer(self):
+ def startIpmiServer(tname, server):
+ self.debug("Starting ipmisim server")
+ try:
+ server.serve_forever()
+ except Exception: pass
+ IpmiServerContext('reset')
+ ThreadedIpmiServer.allow_reuse_address = False
+ server = ThreadedIpmiServer(('0.0.0.0', self.getIpmiServerPort()), IpmiServer)
+ thread.start_new_thread(startIpmiServer, ("ipmi-server", server,))
+ self.server = server
+
+
+ def checkSyncToState(self, state, interval):
+ self.debug("Waiting for background thread to update powerstate to " + state)
+ time.sleep(1 + int(interval)*2/1000) # interval is in ms
+ response = self.getHost(hostId=self.getHost().id).outofbandmanagement
+ self.assertEqual(response.powerstate, state)
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_configure_invalid_driver(self):
+ """
+ Tests out-of-band management configuration with invalid driver
+ """
+ cmd = self.getOobmConfigCmd()
+ cmd.driver = 'randomDriverThatDoesNotExist'
+ try:
+ response = self.apiclient.configureOutOfBandManagement(cmd)
+ self.fail("Expected an exception to be thrown, failing")
+ except Exception: pass
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_configure_default_driver(self):
+ """
+ Tests out-of-band management configuration with valid data
+ """
+ cmd = self.getOobmConfigCmd()
+ response = self.apiclient.configureOutOfBandManagement(cmd)
+ self.assertEqual(response.hostid, cmd.hostid)
+ self.assertEqual(response.driver, cmd.driver)
+ self.assertEqual(response.address, cmd.address)
+ self.assertEqual(response.port, str(cmd.port))
+ self.assertEqual(response.username, cmd.username)
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_enable_feature_invalid(self):
+ """
+ Tests out-of-band management host enable feature with
+ invalid options
+ """
+ cmd = self.getOobmEnableCmd()
+ cmd.hostid = -1
+ try:
+ response = self.apiclient.enableOutOfBandManagementForHost(cmd)
+ self.fail("Expected an exception to be thrown, failing")
+ except Exception: pass
+
+ try:
+ cmd = enableOutOfBandManagementForCluster.enableOutOfBandManagementForClusterCmd()
+ response = self.apiclient.enableOutOfBandManagementForCluster(cmd)
+ self.fail("Expected an exception to be thrown, failing")
+ except Exception: pass
+
+ try:
+ cmd = enableOutOfBandManagementForZone.enableOutOfBandManagementForZoneCmd()
+ response = self.apiclient.enableOutOfBandManagementForZone(cmd)
+ self.fail("Expected an exception to be thrown, failing")
+ except Exception: pass
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_disable_feature_invalid(self):
+ """
+ Tests out-of-band management host disable feature with
+ invalid options
+ """
+ cmd = self.getOobmDisableCmd()
+ cmd.hostid = -1
+ try:
+ response = self.apiclient.disableOutOfBandManagementForHost(cmd)
+ self.fail("Expected an exception to be thrown, failing")
+ except Exception: pass
+
+ try:
+ cmd = disableOutOfBandManagementForCluster.disableOutOfBandManagementForClusterCmd()
+ response = self.apiclient.disableOutOfBandManagementForCluster(cmd)
+ self.fail("Expected an exception to be thrown, failing")
+ except Exception: pass
+
+ try:
+ cmd = disableOutOfBandManagementForZone.disableOutOfBandManagementForZoneCmd()
+ response = self.apiclient.disableOutOfBandManagementForZone(cmd)
+ self.fail("Expected an exception to be thrown, failing")
+ except Exception: pass
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_enable_feature_valid(self):
+ """
+ Tests out-of-band management host enable feature with
+ valid options
+ """
+ self.apiclient.configureOutOfBandManagement(self.getOobmConfigCmd())
+ cmd = self.getOobmEnableCmd()
+ response = self.apiclient.enableOutOfBandManagementForHost(cmd)
+ self.assertEqual(response.hostid, cmd.hostid)
+ self.assertEqual(response.enabled, True)
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_disable_feature_valid(self):
+ """
+ Tests out-of-band management host disable feature with
+ valid options
+ """
+
+ self.apiclient.configureOutOfBandManagement(self.getOobmConfigCmd())
+ cmd = self.getOobmDisableCmd()
+ response = self.apiclient.disableOutOfBandManagementForHost(cmd)
+ self.assertEqual(response.hostid, cmd.hostid)
+ self.assertEqual(response.enabled, False)
+
+ response = self.getHost(hostId=cmd.hostid).outofbandmanagement
+ self.assertEqual(response.powerstate, 'Disabled')
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_enabledisable_across_clusterzones(self):
+ """
+ Tests out-of-band management enable/disable feature at cluster
+ and zone level sequentially Zone > Cluster > Host
+ """
+ self.configureAndEnableOobm()
+ self.startIpmiServer()
+ bmc = IpmiServerContext().bmc
+ bmc.powerstate = 'off'
+
+ host = self.getHost()
+
+ # Disable at zone level
+ cmd = disableOutOfBandManagementForZone.disableOutOfBandManagementForZoneCmd()
+ cmd.zoneid = host.zoneid
+ response = self.apiclient.disableOutOfBandManagementForZone(cmd)
+
+ # Disable at cluster level
+ cmd = disableOutOfBandManagementForCluster.disableOutOfBandManagementForClusterCmd()
+ cmd.clusterid = host.clusterid
+ response = self.apiclient.disableOutOfBandManagementForCluster(cmd)
+
+ # Disable at host level
+ cmd = disableOutOfBandManagementForHost.disableOutOfBandManagementForHostCmd()
+ cmd.hostid = host.id
+ response = self.apiclient.disableOutOfBandManagementForHost(cmd)
+
+ try:
+ self.issuePowerActionCmd('STATUS')
+ self.fail("Exception was expected, oobm is disabled at zone level")
+ except Exception: pass
+
+ # Enable at zone level
+ cmd = enableOutOfBandManagementForZone.enableOutOfBandManagementForZoneCmd()
+ cmd.zoneid = host.zoneid
+ response = self.apiclient.enableOutOfBandManagementForZone(cmd)
+
+ try:
+ self.issuePowerActionCmd('STATUS')
+ self.fail("Exception was expected, oobm is disabled at cluster level")
+ except Exception: pass
+
+ # Check background thread syncs state to Disabled
+ response = self.getHost(hostId=host.id).outofbandmanagement
+ self.assertEqual(response.powerstate, 'Disabled')
+ self.dbclient.execute("update oobm set power_state='On' where port=%d" % self.getIpmiServerPort())
+ interval = list_configurations(
+ self.apiclient,
+ name='outofbandmanagement.sync.interval'
+ )[0].value
+ self.checkSyncToState('Disabled', interval)
+
+ # Enable at cluster level
+ cmd = enableOutOfBandManagementForCluster.enableOutOfBandManagementForClusterCmd()
+ cmd.clusterid = host.clusterid
+ response = self.apiclient.enableOutOfBandManagementForCluster(cmd)
+
+ try:
+ self.issuePowerActionCmd('STATUS')
+ self.fail("Exception was expected, oobm is disabled at host level")
+ except Exception: pass
+
+ # Enable at host level
+ cmd = enableOutOfBandManagementForHost.enableOutOfBandManagementForHostCmd()
+ cmd.hostid = host.id
+ response = self.apiclient.enableOutOfBandManagementForHost(cmd)
+
+ response = self.issuePowerActionCmd('STATUS')
+ self.assertEqual(response.powerstate, 'Off')
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_issue_power_status(self):
+ """
+ Tests out-of-band management issue power action
+ """
+ self.configureAndEnableOobm()
+ self.startIpmiServer()
+ bmc = IpmiServerContext().bmc
+ bmc.powerstate = 'on'
+ response = self.issuePowerActionCmd('STATUS')
+ self.assertEqual(response.powerstate, 'On')
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_issue_power_on(self):
+ """
+ Tests out-of-band management issue power on action
+ """
+ self.configureAndEnableOobm()
+ self.startIpmiServer()
+ self.issuePowerActionCmd('ON')
+ response = self.issuePowerActionCmd('STATUS')
+ self.assertEqual(response.powerstate, 'On')
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_issue_power_off(self):
+ """
+ Tests out-of-band management issue power off action
+ """
+ self.configureAndEnableOobm()
+ self.startIpmiServer()
+ self.issuePowerActionCmd('OFF')
+ response = self.issuePowerActionCmd('STATUS')
+ self.assertEqual(response.powerstate, 'Off')
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_issue_power_cycle(self):
+ """
+ Tests out-of-band management issue power cycle action
+ """
+ self.configureAndEnableOobm()
+ self.startIpmiServer()
+ self.issuePowerActionCmd('CYCLE')
+ response = self.issuePowerActionCmd('STATUS')
+ self.assertEqual(response.powerstate, 'On')
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_issue_power_reset(self):
+ """
+ Tests out-of-band management issue power reset action
+ """
+ self.configureAndEnableOobm()
+ self.startIpmiServer()
+ self.issuePowerActionCmd('RESET')
+ response = self.issuePowerActionCmd('STATUS')
+ self.assertEqual(response.powerstate, 'On')
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_issue_power_soft(self):
+ """
+ Tests out-of-band management issue power soft action
+ """
+ self.configureAndEnableOobm()
+ self.startIpmiServer()
+ self.issuePowerActionCmd('SOFT')
+ response = self.issuePowerActionCmd('STATUS')
+ self.assertEqual(response.powerstate, 'Off')
--- End diff --
The ``test_oobm_issue``* methods seem to be very similar. Have consideration been given to extracting the commonalities to a template method?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61843281
--- Diff: api/src/org/apache/cloudstack/api/command/admin/outofbandmanagement/IssueOutOfBandManagementPowerActionCmd.java ---
@@ -0,0 +1,128 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.api.command.admin.outofbandmanagement;
+
+import com.cloud.event.EventTypes;
+import com.cloud.exception.ConcurrentOperationException;
+import com.cloud.exception.InsufficientCapacityException;
+import com.cloud.exception.NetworkRuleConflictException;
+import com.cloud.exception.ResourceAllocationException;
+import com.cloud.exception.ResourceUnavailableException;
+import com.cloud.host.Host;
+import com.google.common.base.Strings;
+import org.apache.cloudstack.acl.RoleType;
+import org.apache.cloudstack.api.APICommand;
+import org.apache.cloudstack.api.ApiCommandJobType;
+import org.apache.cloudstack.api.ApiConstants;
+import org.apache.cloudstack.api.ApiErrorCode;
+import org.apache.cloudstack.api.BaseAsyncCmd;
+import org.apache.cloudstack.api.Parameter;
+import org.apache.cloudstack.api.ServerApiException;
+import org.apache.cloudstack.api.response.HostResponse;
+import org.apache.cloudstack.api.response.OutOfBandManagementResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagement.PowerOperation;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagementService;
+
+import javax.inject.Inject;
+
+@APICommand(name = "issueOutOfBandManagementPowerAction", description = "Initiates the specified power action to the host's out-of-band management interface",
+ responseObject = OutOfBandManagementResponse.class, requestHasSensitiveInfo = false, responseHasSensitiveInfo = false, authorized = {RoleType.Admin})
+public class IssueOutOfBandManagementPowerActionCmd extends BaseAsyncCmd {
+ @Inject
+ private OutOfBandManagementService outOfBandManagementService;
+
+ /////////////////////////////////////////////////////
+ //////////////// API parameters /////////////////////
+ /////////////////////////////////////////////////////
+
+ @Parameter(name = ApiConstants.HOST_ID, type = CommandType.UUID, entityType = HostResponse.class, required = true, description = "the ID of the host")
+ private Long hostId;
+
+ @Parameter(name = ApiConstants.TIMEOUT, type = CommandType.LONG, description = "optional operation timeout in seconds that overrides the global or cluster-level out-of-band management timeout setting")
+ private Long actionTimeout;
+
+ @Parameter(name = ApiConstants.ACTION, type = CommandType.STRING, required = true, description = "out-of-band management power actions, valid actions are: ON, OFF, CYCLE, RESET, SOFT, STATUS")
+ private String outOfBandManagementPowerOperation;
+
+ /////////////////////////////////////////////////////
+ /////////////// API Implementation///////////////////
+ /////////////////////////////////////////////////////
+
+ @Override
+ public String getCommandName() {
+ return "issueoutofbandmanagementpoweractionresponse";
+ }
+
+ private void validateParams() {
+ if (getHostId() == null || getHostId() < 1L) {
+ throw new ServerApiException(ApiErrorCode.PARAM_ERROR, "Invalid host ID: " + getHostId());
+ }
+ if (Strings.isNullOrEmpty(getOutOfBandManagementPowerOperation())) {
+ throw new ServerApiException(ApiErrorCode.PARAM_ERROR, "Invalid out-of-band management power action: " + getOutOfBandManagementPowerOperation());
+ }
+ }
+
+ @Override
+ public void execute() throws ResourceUnavailableException, InsufficientCapacityException, ServerApiException, ConcurrentOperationException, ResourceAllocationException, NetworkRuleConflictException {
+ validateParams();
+ Host host = _resourceService.getHost(getHostId());
+ if (host == null) {
+ throw new ServerApiException(ApiErrorCode.PARAM_ERROR, "Unable to find host by ID: " + getHostId());
+ }
+ PowerOperation powerOperation = PowerOperation.valueOf(getOutOfBandManagementPowerOperation());
--- End diff --
We're doing Enums.getIfPresent in validateParams therefore a case-insensitive check is not needed here, if validate does not fail PowerOperation.valueOf will always return an valid enum.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61815152
--- Diff: test/integration/smoke/test_outofbandmanagement.py ---
@@ -0,0 +1,561 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import marvin
+from marvin.cloudstackTestCase import *
+from marvin.cloudstackAPI import *
+from marvin.lib.utils import *
+from marvin.lib.base import *
+from marvin.lib.common import *
+from marvin.lib.utils import (random_gen)
+from nose.plugins.attrib import attr
+
+from ipmisim.ipmisim import IpmiServerContext, IpmiServer, ThreadedIpmiServer
+
+import socket
+import sys
+import thread
+import time
+
+
+class TestOutOfBandManagement(cloudstackTestCase):
+ """ Test cases for out of band management
+ """
+
+ def setUp(self):
+ self.apiclient = self.testClient.getApiClient()
+ self.hypervisor = self.testClient.getHypervisorInfo()
+ self.dbclient = self.testClient.getDbConnection()
+ self.services = self.testClient.getParsedTestDataConfig()
+ self.mgtSvrDetails = self.config.__dict__["mgtSvr"][0].__dict__
+
+ self.zone = get_zone(self.apiclient, self.testClient.getZoneForTests())
+ self.host = None
+ self.server = None
+
+ # use random port for ipmisim
+ s = socket.socket()
+ s.bind(('', 0))
+ self.serverPort = s.getsockname()[1]
+ s.close()
+
+ self.cleanup = []
+
+
+ def tearDown(self):
+ try:
+ self.dbclient.execute("delete from oobm where port=%d" % self.getIpmiServerPort())
+ self.dbclient.execute("delete from mshost_peer where peer_runid=%s" % self.getFakeMsRunId())
+ self.dbclient.execute("delete from mshost where runid=%s" % self.getFakeMsRunId())
+ self.dbclient.execute("delete from cluster_details where name='outOfBandManagementEnabled'")
+ self.dbclient.execute("delete from data_center_details where name='outOfBandManagementEnabled'")
+ cleanup_resources(self.apiclient, self.cleanup)
+ if self.server:
+ self.server.shutdown()
+ self.server.server_close()
+ except Exception as e:
+ raise Exception("Warning: Exception during cleanup : %s" % e)
+
+
+ def getFakeMsId(self):
+ return 1234567890
+
+
+ def getFakeMsRunId(self):
+ return 123456
--- End diff --
Consider using a random value to allow multiple executions of this test case to occur without impacting each other in the event that the cleanup does not complete successfully.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-214637482
>@rhtyd maybe in a follow-up; can we split the test in hardware required true and false?
@DaanHoogland for this feature's integration test, we don't need an explicit hardware required arg since we launch an ipmi simulator using `ipmisim` within the test wherever needed to perform end to end ipmi/power-action testing. If I still to explicitly put the value, it would be `False` on all test cases within `test_outofbandmanagement.py`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61778988
--- Diff: api/src/org/apache/cloudstack/outofbandmanagement/OutOfBandManagementService.java ---
@@ -0,0 +1,51 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.outofbandmanagement;
+
+import com.cloud.dc.DataCenter;
+import com.cloud.host.Host;
+import com.cloud.org.Cluster;
+import com.google.common.collect.ImmutableMap;
+import org.apache.cloudstack.api.response.OutOfBandManagementResponse;
+import org.apache.cloudstack.framework.config.ConfigKey;
+
+import java.util.List;
+
+public interface OutOfBandManagementService {
+
+ ConfigKey<Long> OutOfBandManagementActionTimeout = new ConfigKey<Long>("Advanced", Long.class, "outofbandmanagement.action.timeout", "60",
+ "The out of band management action timeout in seconds, configurable by cluster", true, ConfigKey.Scope.Cluster);
+
+ ConfigKey<Long> OutOfBandManagementSyncThreadInterval = new ConfigKey<Long>("Advanced", Long.class, "outofbandmanagement.sync.interval", "300000",
+ "The interval (in milliseconds) when the out-of-band management background sync are retrieved", true, ConfigKey.Scope.Global);
+
+ ConfigKey<Integer> OutOfBandManagementSyncThreadPoolSize = new ConfigKey<Integer>("Advanced", Integer.class, "outofbandmanagement.sync.poolsize", "50",
--- End diff --
Since these constants are in the namespace of the ``OutOfBandManagementService``, why not remove ``OutOfManagement`` prefix from these constants?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-218944171
@swill thanks, will have a look at it
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r62616616
--- Diff: utils/src/main/java/org/apache/cloudstack/utils/process/ProcessRunner.java ---
@@ -0,0 +1,112 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+
+package org.apache.cloudstack.utils.process;
+
+import com.cloud.utils.concurrency.NamedThreadFactory;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Strings;
+import org.apache.log4j.Logger;
+import org.joda.time.Duration;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.List;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+public class ProcessRunner {
+ public static final Logger LOG = Logger.getLogger(ProcessRunner.class);
+
+ private static final ExecutorService processExecutor = Executors.newCachedThreadPool(new NamedThreadFactory("ProcessRunner"));
+
+ private static String readStream(final InputStream inputStream) {
+ String text = null;
+ try {
+ final BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(inputStream));
+ String line;
+ while ((line = bufferedReader.readLine()) != null) {
+ if (Strings.isNullOrEmpty(text)) {
+ text = line;
+ } else {
+ text = text + "\n" + line;
+ }
+ }
+ } catch (IOException e) {
--- End diff --
Thrown
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r62570929
--- Diff: utils/src/main/java/org/apache/cloudstack/utils/process/ProcessRunner.java ---
@@ -0,0 +1,112 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+
+package org.apache.cloudstack.utils.process;
+
+import com.cloud.utils.concurrency.NamedThreadFactory;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Strings;
+import org.apache.log4j.Logger;
+import org.joda.time.Duration;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.List;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+public class ProcessRunner {
+ public static final Logger LOG = Logger.getLogger(ProcessRunner.class);
+
+ private static final ExecutorService processExecutor = Executors.newCachedThreadPool(new NamedThreadFactory("ProcessRunner"));
+
+ private static String readStream(final InputStream inputStream) {
+ String text = null;
+ try {
+ final BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(inputStream));
+ String line;
+ while ((line = bufferedReader.readLine()) != null) {
+ if (Strings.isNullOrEmpty(text)) {
+ text = line;
+ } else {
+ text = text + "\n" + line;
+ }
+ }
+ } catch (IOException e) {
+ if (LOG.isTraceEnabled()) {
+ LOG.trace("ProcessRunner::readStream failed due to: " + e.getMessage());
+ }
+ }
+ return text;
+ }
+
+ public static ProcessResult executeCommands(final List<String> commands, final Duration timeOut) {
+ Preconditions.checkArgument(commands != null && timeOut != null);
+
+ int retVal = -2;
+ String stdOutput = null;
+ String stdError = null;
+
+ try {
+ final Process process = new ProcessBuilder().command(commands).start();
+ if (timeOut.getStandardSeconds() > 0) {
+ final Future<Integer> processFuture = processExecutor.submit(new Callable<Integer>() {
+ @Override
+ public Integer call() throws Exception {
+ return process.waitFor();
+ }
+ });
+ try {
+ retVal = processFuture.get(timeOut.getStandardSeconds(), TimeUnit.SECONDS);
+ } catch (ExecutionException | TimeoutException e) {
+ retVal = -1;
+ stdError = "Operation timed out, aborted";
+ if (LOG.isTraceEnabled()) {
+ LOG.trace("Failed to complete the requested command within timeout: " + e.getMessage());
+ }
+ } finally {
+ if (Strings.isNullOrEmpty(stdError)) {
+ stdOutput = readStream(process.getInputStream());
+ stdError = readStream(process.getErrorStream());
+ }
+ }
+ } else {
+ retVal = process.waitFor();
+ stdOutput = readStream(process.getInputStream());
+ stdError = readStream(process.getErrorStream());
+ }
+ process.destroy();
--- End diff --
Should this call be part of a ``finally`` block to ensure cleanup in all circumstances?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r62569447
--- Diff: utils/src/main/java/org/apache/cloudstack/utils/process/ProcessRunner.java ---
@@ -0,0 +1,112 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+
+package org.apache.cloudstack.utils.process;
+
+import com.cloud.utils.concurrency.NamedThreadFactory;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Strings;
+import org.apache.log4j.Logger;
+import org.joda.time.Duration;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.List;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+public class ProcessRunner {
+ public static final Logger LOG = Logger.getLogger(ProcessRunner.class);
+
+ private static final ExecutorService processExecutor = Executors.newCachedThreadPool(new NamedThreadFactory("ProcessRunner"));
+
+ private static String readStream(final InputStream inputStream) {
+ String text = null;
+ try {
+ final BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(inputStream));
+ String line;
+ while ((line = bufferedReader.readLine()) != null) {
+ if (Strings.isNullOrEmpty(text)) {
+ text = line;
+ } else {
+ text = text + "\n" + line;
+ }
+ }
+ } catch (IOException e) {
--- End diff --
Why don't we re-throw this exception as an unchecked exception? Failing to read the standard out and error streams seems like it would represent a failure of the operation.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r62568855
--- Diff: utils/src/main/java/org/apache/cloudstack/utils/process/ProcessRunner.java ---
@@ -0,0 +1,112 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+
+package org.apache.cloudstack.utils.process;
+
+import com.cloud.utils.concurrency.NamedThreadFactory;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Strings;
+import org.apache.log4j.Logger;
+import org.joda.time.Duration;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.List;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+public class ProcessRunner {
+ public static final Logger LOG = Logger.getLogger(ProcessRunner.class);
+
+ private static final ExecutorService processExecutor = Executors.newCachedThreadPool(new NamedThreadFactory("ProcessRunner"));
+
+ private static String readStream(final InputStream inputStream) {
+ String text = null;
+ try {
+ final BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(inputStream));
+ String line;
+ while ((line = bufferedReader.readLine()) != null) {
+ if (Strings.isNullOrEmpty(text)) {
+ text = line;
+ } else {
+ text = text + "\n" + line;
+ }
+ }
+ } catch (IOException e) {
+ if (LOG.isTraceEnabled()) {
+ LOG.trace("ProcessRunner::readStream failed due to: " + e.getMessage());
+ }
+ }
+ return text;
+ }
+
+ public static ProcessResult executeCommands(final List<String> commands, final Duration timeOut) {
+ Preconditions.checkArgument(commands != null && timeOut != null);
+
+ int retVal = -2;
+ String stdOutput = null;
+ String stdError = null;
+
+ try {
+ final Process process = new ProcessBuilder().command(commands).start();
+ if (timeOut.getStandardSeconds() > 0) {
--- End diff --
Should a command without a timeout be accepted?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by swill <gi...@git.apache.org>.
Github user swill commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-216309958
@jburwell no worries, I know how that goes. :) Thanks for your support regardless...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r62612076
--- Diff: utils/src/main/java/org/apache/cloudstack/utils/process/ProcessRunner.java ---
@@ -0,0 +1,112 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+
+package org.apache.cloudstack.utils.process;
+
+import com.cloud.utils.concurrency.NamedThreadFactory;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Strings;
+import org.apache.log4j.Logger;
+import org.joda.time.Duration;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.List;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+public class ProcessRunner {
+ public static final Logger LOG = Logger.getLogger(ProcessRunner.class);
+
+ private static final ExecutorService processExecutor = Executors.newCachedThreadPool(new NamedThreadFactory("ProcessRunner"));
+
+ private static String readStream(final InputStream inputStream) {
+ String text = null;
+ try {
+ final BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(inputStream));
+ String line;
+ while ((line = bufferedReader.readLine()) != null) {
+ if (Strings.isNullOrEmpty(text)) {
+ text = line;
+ } else {
+ text = text + "\n" + line;
+ }
+ }
+ } catch (IOException e) {
+ if (LOG.isTraceEnabled()) {
+ LOG.trace("ProcessRunner::readStream failed due to: " + e.getMessage());
+ }
+ }
+ return text;
+ }
+
+ public static ProcessResult executeCommands(final List<String> commands, final Duration timeOut) {
+ Preconditions.checkArgument(commands != null && timeOut != null);
+
+ int retVal = -2;
+ String stdOutput = null;
+ String stdError = null;
+
+ try {
+ final Process process = new ProcessBuilder().command(commands).start();
+ if (timeOut.getStandardSeconds() > 0) {
+ final Future<Integer> processFuture = processExecutor.submit(new Callable<Integer>() {
+ @Override
+ public Integer call() throws Exception {
+ return process.waitFor();
+ }
+ });
+ try {
+ retVal = processFuture.get(timeOut.getStandardSeconds(), TimeUnit.SECONDS);
+ } catch (ExecutionException | TimeoutException e) {
--- End diff --
Will split the catch cases separately.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61844447
--- Diff: api/src/org/apache/cloudstack/outofbandmanagement/OutOfBandManagementService.java ---
@@ -0,0 +1,51 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.outofbandmanagement;
+
+import com.cloud.dc.DataCenter;
+import com.cloud.host.Host;
+import com.cloud.org.Cluster;
+import com.google.common.collect.ImmutableMap;
+import org.apache.cloudstack.api.response.OutOfBandManagementResponse;
+import org.apache.cloudstack.framework.config.ConfigKey;
+
+import java.util.List;
+
+public interface OutOfBandManagementService {
+
+ ConfigKey<Long> OutOfBandManagementActionTimeout = new ConfigKey<Long>("Advanced", Long.class, "outofbandmanagement.action.timeout", "60",
+ "The out of band management action timeout in seconds, configurable by cluster", true, ConfigKey.Scope.Cluster);
+
+ ConfigKey<Long> OutOfBandManagementSyncThreadInterval = new ConfigKey<Long>("Advanced", Long.class, "outofbandmanagement.sync.interval", "300000",
+ "The interval (in milliseconds) when the out-of-band management background sync are retrieved", true, ConfigKey.Scope.Global);
+
+ ConfigKey<Integer> OutOfBandManagementSyncThreadPoolSize = new ConfigKey<Integer>("Advanced", Integer.class, "outofbandmanagement.sync.poolsize", "50",
--- End diff --
Fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61810223
--- Diff: server/src/org/apache/cloudstack/outofbandmanagement/OutOfBandManagementServiceImpl.java ---
@@ -0,0 +1,532 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.outofbandmanagement;
+
+import com.cloud.alert.AlertManager;
+import com.cloud.dc.ClusterDetailsDao;
+import com.cloud.dc.ClusterDetailsVO;
+import com.cloud.dc.DataCenter;
+import com.cloud.dc.DataCenterDetailVO;
+import com.cloud.dc.dao.DataCenterDetailsDao;
+import com.cloud.domain.Domain;
+import com.cloud.event.ActionEvent;
+import com.cloud.event.ActionEventUtils;
+import com.cloud.event.EventTypes;
+import com.cloud.host.Host;
+import com.cloud.host.dao.HostDao;
+import com.cloud.org.Cluster;
+import com.cloud.utils.component.Manager;
+import com.cloud.utils.component.ManagerBase;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.db.Transaction;
+import com.cloud.utils.db.TransactionCallback;
+import com.cloud.utils.db.TransactionStatus;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.fsm.NoTransitionException;
+import com.google.common.base.Strings;
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import com.google.common.collect.ImmutableMap;
+import org.apache.cloudstack.api.response.OutOfBandManagementResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.framework.config.ConfigKey;
+import org.apache.cloudstack.framework.config.Configurable;
+import org.apache.cloudstack.outofbandmanagement.dao.OutOfBandManagementDao;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverChangePasswordCommand;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverPowerCommand;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverResponse;
+import org.apache.cloudstack.utils.identity.ManagementServerNode;
+import org.apache.log4j.Logger;
+import org.springframework.stereotype.Component;
+
+import javax.ejb.Local;
+import javax.inject.Inject;
+import javax.naming.ConfigurationException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+@Component
+@Local(value = {OutOfBandManagementService.class})
+public class OutOfBandManagementServiceImpl extends ManagerBase implements OutOfBandManagementService, Manager, Configurable {
+ public static final Logger LOG = Logger.getLogger(OutOfBandManagementServiceImpl.class);
+
+ @Inject
+ private ClusterDetailsDao clusterDetailsDao;
+ @Inject
+ private DataCenterDetailsDao dataCenterDetailsDao;
+ @Inject
+ private OutOfBandManagementDao outOfBandManagementDao;
+ @Inject
+ private HostDao hostDao;
+ @Inject
+ private AlertManager alertMgr;
+
+ private String name;
+ private long serviceId;
+
+ private List<OutOfBandManagementDriver> outOfBandManagementDrivers = new ArrayList<>();
+ private Map<String, OutOfBandManagementDriver> outOfBandManagementDriversMap = new HashMap<String, OutOfBandManagementDriver>();
+
+ private static final String OOBM_ENABLED_DETAIL = "outOfBandManagementEnabled";
+ private static final int ACQUIRE_GLOBAL_LOCK_TIMEOUT_FOR_HOST = 120;
+
+ private Cache<Long, Long> hostAlertCache;
+ private static ExecutorService backgroundSyncExecutor;
+
+ private String getOutOfBandManagementHostLock(long id) {
+ return "oobm.host." + id;
+ }
+
+ private void initializeDriversMap() {
+ if (outOfBandManagementDriversMap.isEmpty() && outOfBandManagementDrivers != null && outOfBandManagementDrivers.size() > 0) {
+ for (final OutOfBandManagementDriver driver : outOfBandManagementDrivers) {
+ outOfBandManagementDriversMap.put(driver.getName().toLowerCase(), driver);
+ }
+ LOG.debug("Discovered out-of-band management drivers configured in the OutOfBandManagementService");
+ }
+ }
+
+ private OutOfBandManagementDriver getDriver(final OutOfBandManagement outOfBandManagementConfig) {
+ if (!Strings.isNullOrEmpty(outOfBandManagementConfig.getDriver())) {
+ final OutOfBandManagementDriver driver = outOfBandManagementDriversMap.get(outOfBandManagementConfig.getDriver());
+ if (driver != null) {
+ return driver;
+ }
+ }
+ throw new CloudRuntimeException("Configured out-of-band management driver is not available. Aborting any out-of-band management action.");
+ }
+
+ protected OutOfBandManagement updateConfig(final OutOfBandManagement outOfBandManagementConfig, final ImmutableMap<OutOfBandManagement.Option, String> options) {
+ if (outOfBandManagementConfig == null) {
+ throw new CloudRuntimeException("Out-of-band management is not configured for the host. Aborting.");
+ }
+ if (options == null) {
+ return outOfBandManagementConfig;
+ }
+ for (OutOfBandManagement.Option option: options.keySet()) {
+ final String value = options.get(option);
+ if (Strings.isNullOrEmpty(value)) {
+ continue;
+ }
+ switch (option) {
+ case DRIVER:
+ outOfBandManagementConfig.setDriver(value);
+ break;
+ case ADDRESS:
+ outOfBandManagementConfig.setAddress(value);
+ break;
+ case PORT:
+ outOfBandManagementConfig.setPort(Integer.parseInt(value));
+ break;
+ case USERNAME:
+ outOfBandManagementConfig.setUsername(value);
+ break;
+ case PASSWORD:
+ outOfBandManagementConfig.setPassword(value);
+ break;
+ }
+ }
+ return outOfBandManagementConfig;
+ }
+
+ protected ImmutableMap<OutOfBandManagement.Option, String> getOptions(final OutOfBandManagement outOfBandManagementConfig) {
+ final ImmutableMap.Builder<OutOfBandManagement.Option, String> optionsBuilder = ImmutableMap.builder();
+ if (outOfBandManagementConfig == null) {
+ throw new CloudRuntimeException("Out-of-band management is not configured for the host. Aborting.");
+ }
+ for (OutOfBandManagement.Option option: OutOfBandManagement.Option.values()) {
+ String value = null;
+ switch (option) {
+ case DRIVER:
+ value = outOfBandManagementConfig.getDriver();
+ break;
+ case ADDRESS:
+ value = outOfBandManagementConfig.getAddress();
+ break;
+ case PORT:
+ if (outOfBandManagementConfig.getPort() != null) {
+ value = String.valueOf(outOfBandManagementConfig.getPort());
+ }
+ break;
+ case USERNAME:
+ value = outOfBandManagementConfig.getUsername();
+ break;
+ case PASSWORD:
+ value = outOfBandManagementConfig.getPassword();
+ break;
+ }
+ if (value != null) {
+ optionsBuilder.put(option, value);
+ }
+ }
+ return optionsBuilder.build();
+ }
+
+ private void sendAuthError(final Host host, final String message) {
+ try {
+ hostAlertCache.asMap().putIfAbsent(host.getId(), 0L);
+ Long sentCount = hostAlertCache.asMap().get(host.getId());
+ if (sentCount != null && sentCount <= 0) {
+ boolean concurrentUpdateResult = hostAlertCache.asMap().replace(host.getId(), sentCount, sentCount+1L);
+ if (concurrentUpdateResult) {
+ final String subject = String.format("Out-of-band management auth-error detected for host:%d in cluster:%d, zone:%d", host.getId(), host.getClusterId(), host.getDataCenterId());
+ LOG.error(subject + ": " + message);
+ alertMgr.sendAlert(AlertManager.AlertType.ALERT_TYPE_OOBM_AUTH_ERROR, host.getDataCenterId(), host.getPodId(), subject, message);
+ }
+ }
+ } catch (Exception ignored) {
+ }
+ }
+
+ private boolean transitionPowerState(OutOfBandManagement.PowerState.Event event, OutOfBandManagement outOfBandManagementHost) {
+ if (outOfBandManagementHost == null) {
+ return false;
+ }
+ OutOfBandManagement.PowerState currentPowerState = outOfBandManagementHost.getPowerState();
+ try {
+ OutOfBandManagement.PowerState newPowerState = OutOfBandManagement.PowerState.getStateMachine().getNextState(currentPowerState, event);
+ boolean result = outOfBandManagementDao.updateState(currentPowerState, event, newPowerState, outOfBandManagementHost, null);
+ if (result) {
+ final String message = String.format("Transitioned out-of-band management power state from:%s to:%s due to event:%s for the host id:%d", currentPowerState, newPowerState, event, outOfBandManagementHost.getHostId());
+ LOG.debug(message);
+ ActionEventUtils.onActionEvent(CallContext.current().getCallingUserId(), CallContext.current().getCallingAccountId(), Domain.ROOT_DOMAIN,
+ EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_POWERSTATE_TRANSITION, message);
+ }
+ return result;
+ } catch (NoTransitionException ignored) {
+ LOG.trace(String.format("Unable to transition out-of-band management power state for host id=%s for the event=%s and current power state=%s", outOfBandManagementHost.getHostId(), event, currentPowerState));
+ }
+ return false;
+ }
+
+ private boolean isOutOfBandManagementEnabledForZone(Long zoneId) {
+ if (zoneId == null) {
+ return true;
+ }
+ final DataCenterDetailVO zoneDetails = dataCenterDetailsDao.findDetail(zoneId, OOBM_ENABLED_DETAIL);
+ if (zoneDetails != null && !Strings.isNullOrEmpty(zoneDetails.getValue()) && !Boolean.valueOf(zoneDetails.getValue())) {
+ return false;
+ }
+ return true;
+ }
+
+ private boolean isOutOfBandManagementEnabledForCluster(Long clusterId) {
+ if (clusterId == null) {
+ return true;
+ }
+ final ClusterDetailsVO clusterDetails = clusterDetailsDao.findDetail(clusterId, OOBM_ENABLED_DETAIL);
+ if (clusterDetails != null && !Strings.isNullOrEmpty(clusterDetails.getValue()) && !Boolean.valueOf(clusterDetails.getValue())) {
+ return false;
+ }
+ return true;
+ }
+
+ private boolean isOutOfBandManagementEnabledForHost(Long hostId) {
+ if (hostId == null) {
+ return false;
+ }
+ final OutOfBandManagement outOfBandManagementConfig = outOfBandManagementDao.findByHost(hostId);
+ if (outOfBandManagementConfig == null || !outOfBandManagementConfig.isEnabled()) {
+ return false;
+ }
+ return true;
+ }
+
+ private void checkOutOfBandManagementEnabledByZoneClusterHost(final Host host) {
+ if (!isOutOfBandManagementEnabledForZone(host.getDataCenterId())) {
+ throw new CloudRuntimeException("Out-of-band management is disabled for the host's zone. Aborting Operation.");
+ }
+ if (!isOutOfBandManagementEnabledForCluster(host.getClusterId())) {
+ throw new CloudRuntimeException("Out-of-band management is disabled for the host's cluster. Aborting Operation.");
+ }
+ if (!isOutOfBandManagementEnabledForHost(host.getId())) {
+ throw new CloudRuntimeException("Out-of-band management is disabled or not configured for the host. Aborting Operation.");
+ }
+ }
+
+ public boolean isOutOfBandManagementEnabled(final Host host) {
+ return isOutOfBandManagementEnabledForZone(host.getDataCenterId())
+ && isOutOfBandManagementEnabledForCluster(host.getClusterId())
+ && isOutOfBandManagementEnabledForHost(host.getId());
+ }
+
+ public boolean transitionPowerStateToDisabled(List<? extends Host> hosts) {
+ boolean result = true;
+ for (Host host : hosts) {
+ result = result && transitionPowerState(OutOfBandManagement.PowerState.Event.Disabled,
+ outOfBandManagementDao.findByHost(host.getId()));
+ }
+ return result;
+ }
+
+ public void submitBackgroundPowerSyncTask(final Host host) {
+ if (host != null) {
+ // Note: This is a blocking queue based executor
+ backgroundSyncExecutor.submit(new OutOfBandManagementBackgroundTask(this, host, OutOfBandManagement.PowerOperation.STATUS));
+ }
+ }
+
+ private OutOfBandManagementResponse buildEnableDisableResponse(final boolean enabled) {
+ final OutOfBandManagementResponse response = new OutOfBandManagementResponse();
+ response.setEnabled(enabled);
+ response.setSuccess(true);
+ return response;
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_ENABLEDISABLE, eventDescription = "enabling/disabling out-of-band management on a zone")
+ public OutOfBandManagementResponse enableDisableOutOfBandManagement(final DataCenter zone, final boolean enabled) {
+ dataCenterDetailsDao.persist(zone.getId(), OOBM_ENABLED_DETAIL, String.valueOf(enabled));
+ if (!enabled) {
+ transitionPowerStateToDisabled(hostDao.findByDataCenterId(zone.getId()));
+ }
+ return buildEnableDisableResponse(enabled);
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_ENABLEDISABLE, eventDescription = "enabling/disabling out-of-band management on a cluster")
+ public OutOfBandManagementResponse enableDisableOutOfBandManagement(final Cluster cluster, final boolean enabled) {
+ clusterDetailsDao.persist(cluster.getId(), OOBM_ENABLED_DETAIL, String.valueOf(enabled));
+ if (!enabled) {
+ transitionPowerStateToDisabled(hostDao.findByClusterId(cluster.getId()));
+ }
+ return buildEnableDisableResponse(enabled);
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_ENABLEDISABLE, eventDescription = "enabling/disabling out-of-band management on a host")
+ public OutOfBandManagementResponse enableDisableOutOfBandManagement(final Host host, final boolean enabled) {
+ final OutOfBandManagement outOfBandManagementConfig = outOfBandManagementDao.findByHost(host.getId());
+ if (outOfBandManagementConfig == null) {
+ final OutOfBandManagementResponse response = new OutOfBandManagementResponse(null);
+ response.setSuccess(false);
+ response.setResultDescription("Out-of-band management is not configured for the host. Please configure the host before enabling/disabling it.");
+ return response;
+ }
+ hostAlertCache.invalidate(host.getId());
+ outOfBandManagementConfig.setEnabled(enabled);
+ boolean updateResult = outOfBandManagementDao.update(outOfBandManagementConfig.getId(), (OutOfBandManagementVO) outOfBandManagementConfig);
+ if (updateResult && !enabled) {
+ transitionPowerStateToDisabled(Collections.singletonList(host));
+ }
+ return buildEnableDisableResponse(enabled && updateResult);
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_CONFIGURE, eventDescription = "updating out-of-band management configuration")
+ public OutOfBandManagementResponse configureOutOfBandManagement(final Host host, final ImmutableMap<OutOfBandManagement.Option, String> options) {
+ OutOfBandManagement outOfBandManagementConfig = outOfBandManagementDao.findByHost(host.getId());
+ if (outOfBandManagementConfig == null) {
+ outOfBandManagementConfig = outOfBandManagementDao.persist(new OutOfBandManagementVO(host.getId()));
+ }
+ outOfBandManagementConfig = updateConfig(outOfBandManagementConfig, options);
+ if (Strings.isNullOrEmpty(outOfBandManagementConfig.getDriver()) || !outOfBandManagementDriversMap.containsKey(outOfBandManagementConfig.getDriver().toLowerCase())) {
+ throw new CloudRuntimeException("Out-of-band management driver is not available. Please provide a valid driver name.");
+ }
+
+ boolean updatedConfig = outOfBandManagementDao.update(outOfBandManagementConfig.getId(), (OutOfBandManagementVO) outOfBandManagementConfig);
+ CallContext.current().setEventDetails("host id:" + host.getId() + " configuration:" + outOfBandManagementConfig.getAddress() + ":" + outOfBandManagementConfig.getPort());
+
+ if (!updatedConfig) {
+ throw new CloudRuntimeException("Failed to update out-of-band management config for the host in the database.");
+ }
+
+ String result = "Out-of-band management successfully configured for the host";
+ LOG.debug(result);
+
+ final OutOfBandManagementResponse response = new OutOfBandManagementResponse(outOfBandManagementDao.findByHost(host.getId()));
+ response.setResultDescription(result);
+ response.setSuccess(true);
+ return response;
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_ACTION, eventDescription = "issuing Host out-of-band management action", async = true)
+ public OutOfBandManagementResponse executeOutOfBandManagementPowerOperation(final Host host, final OutOfBandManagement.PowerOperation powerOperation, final Long timeout) {
+ checkOutOfBandManagementEnabledByZoneClusterHost(host);
+ final OutOfBandManagement outOfBandManagementConfig = outOfBandManagementDao.findByHost(host.getId());
+ final ImmutableMap<OutOfBandManagement.Option, String> options = getOptions(outOfBandManagementConfig);
+ final OutOfBandManagementDriver driver = getDriver(outOfBandManagementConfig);
+
+ Long actionTimeOut = timeout;
+ if (actionTimeOut == null) {
+ actionTimeOut = OutOfBandManagementActionTimeout.valueIn(host.getClusterId());
+ }
+
+ final OutOfBandManagementDriverPowerCommand cmd = new OutOfBandManagementDriverPowerCommand(options, actionTimeOut, powerOperation);
+ final OutOfBandManagementDriverResponse driverResponse = driver.execute(cmd);
+
+ if (driverResponse == null) {
+ throw new CloudRuntimeException(String.format("Out-of-band Management action (%s) on host (%s) failed due to no response from the driver", powerOperation, host.getUuid()));
+ }
+
+ if (powerOperation.equals(OutOfBandManagement.PowerOperation.STATUS)) {
+ transitionPowerState(driverResponse.toEvent(), outOfBandManagementConfig);
+ }
+
+ if (!driverResponse.isSuccess()) {
+ String errorMessage = String.format("Out-of-band Management action (%s) on host (%s) failed with error: %s", powerOperation, host.getUuid(), driverResponse.getError());
+ if (driverResponse.hasAuthFailure()) {
+ errorMessage = String.format("Out-of-band Management action (%s) on host (%s) failed due to authentication error: %s. Please check configured credentials.", powerOperation, host.getUuid(), driverResponse.getError());
+ sendAuthError(host, errorMessage);
+ }
+ if (!powerOperation.equals(OutOfBandManagement.PowerOperation.STATUS)) {
+ LOG.debug(errorMessage);
+ }
+ throw new CloudRuntimeException(errorMessage);
+ }
+
+ final OutOfBandManagementResponse response = new OutOfBandManagementResponse(outOfBandManagementDao.findByHost(host.getId()));
+ response.setSuccess(driverResponse.isSuccess());
+ response.setResultDescription(driverResponse.getResult());
+ response.setId(host.getUuid());
+ response.setOutOfBandManagementAction(powerOperation.toString());
+ return response;
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_CHANGE_PASSWORD, eventDescription = "updating out-of-band management password")
+ public OutOfBandManagementResponse changeOutOfBandManagementPassword(final Host host, final String newPassword) {
+ checkOutOfBandManagementEnabledByZoneClusterHost(host);
+ if (Strings.isNullOrEmpty(newPassword)) {
+ throw new CloudRuntimeException(String.format("Cannot change out-of-band management password as provided new-password is null or empty for the host %s.", host.getUuid()));
+ }
+ GlobalLock outOfBandManagementHostLock = GlobalLock.getInternLock(getOutOfBandManagementHostLock(host.getId()));
+ try {
+ if (outOfBandManagementHostLock.lock(ACQUIRE_GLOBAL_LOCK_TIMEOUT_FOR_HOST)) {
+ try {
--- End diff --
Why is this nested ``try`` necessary? It seems that the enclosing ``try`` block would be sufficient.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61789411
--- Diff: plugins/outofbandmanagement-drivers/ipmitool/src/org/apache/cloudstack/outofbandmanagement/driver/ipmitool/IpmitoolWrapper.java ---
@@ -0,0 +1,166 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.outofbandmanagement.driver.ipmitool;
+
+import com.cloud.utils.StringUtils;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.google.common.base.Strings;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagement;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverResponse;
+import org.apache.cloudstack.utils.process.ProcessRunner;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.List;
+
+public class IpmitoolWrapper {
+ public static final Logger LOG = Logger.getLogger(IpmitoolWrapper.class);
+
+ public static String parsePowerCommand(OutOfBandManagement.PowerOperation operation) {
+ if (operation == null) {
+ throw new IllegalStateException("Invalid power operation requested");
+ }
+ switch (operation) {
+ case ON:
+ case OFF:
+ case CYCLE:
+ case RESET:
+ case SOFT:
+ case STATUS:
+ break;
+ default:
+ throw new IllegalStateException("Invalid power operation requested");
+ }
+ return operation.toString().toLowerCase();
+ }
+
+ public static OutOfBandManagement.PowerState parsePowerState(final String standardOutput) {
+ if (Strings.isNullOrEmpty(standardOutput)) {
+ return OutOfBandManagement.PowerState.Unknown;
+ }
+ if (standardOutput.equals("Chassis Power is on")) {
+ return OutOfBandManagement.PowerState.On;
+ } else if (standardOutput.equals("Chassis Power is off")) {
+ return OutOfBandManagement.PowerState.Off;
+ }
+ return OutOfBandManagement.PowerState.Unknown;
+ }
+
+ public static List<String> getIpmiToolCommandArgs(final String ipmiToolPath, final String ipmiInterface, final String retries,
+ final ImmutableMap<OutOfBandManagement.Option, String> options, String... commands) {
+
+ ImmutableList.Builder<String> ipmiToolCommands = ImmutableList.<String>builder()
+ .add(ipmiToolPath)
+ .add("-I")
+ .add(ipmiInterface)
+ .add("-R")
+ .add(retries)
+ .add("-v");
+
+ if (options != null) {
+ for (ImmutableMap.Entry<OutOfBandManagement.Option, String> option : options.entrySet()) {
+ switch (option.getKey()) {
+ case ADDRESS:
+ ipmiToolCommands.add("-H");
+ break;
+ case PORT:
+ ipmiToolCommands.add("-p");
+ break;
+ case USERNAME:
+ ipmiToolCommands.add("-U");
+ break;
+ case PASSWORD:
+ ipmiToolCommands.add("-P");
+ break;
+ default:
+ continue;
+ }
+ ipmiToolCommands.add(option.getValue());
+ }
+ }
+ for (String command : commands) {
+ ipmiToolCommands.add(command);
+ }
+ return ipmiToolCommands.build();
+ }
+
+ public static String findIpmiUser(final String usersList, final String username) {
+ // Expected usersList string contains legends on first line and users on rest
+ // ID Name Callin Link Auth IPMI Msg Channel Priv Limit
+ // 1 admin true true true ADMINISTRATOR
--- End diff --
Move this documentation to Javadoc for the method.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61777268
--- Diff: api/src/org/apache/cloudstack/api/command/admin/outofbandmanagement/EnableOutOfBandManagementForZoneCmd.java ---
@@ -0,0 +1,105 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.api.command.admin.outofbandmanagement;
+
+import com.cloud.dc.DataCenter;
+import com.cloud.event.EventTypes;
+import com.cloud.exception.ConcurrentOperationException;
+import com.cloud.exception.InsufficientCapacityException;
+import com.cloud.exception.NetworkRuleConflictException;
+import com.cloud.exception.ResourceAllocationException;
+import com.cloud.exception.ResourceUnavailableException;
+import org.apache.cloudstack.acl.RoleType;
+import org.apache.cloudstack.api.APICommand;
+import org.apache.cloudstack.api.ApiConstants;
+import org.apache.cloudstack.api.ApiErrorCode;
+import org.apache.cloudstack.api.BaseAsyncCmd;
+import org.apache.cloudstack.api.BaseCmd;
+import org.apache.cloudstack.api.Parameter;
+import org.apache.cloudstack.api.ServerApiException;
+import org.apache.cloudstack.api.response.OutOfBandManagementResponse;
+import org.apache.cloudstack.api.response.ZoneResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagementService;
+
+import javax.inject.Inject;
+
+@APICommand(name = "enableOutOfBandManagementForZone", description = "Enables out-of-band management for a zone",
--- End diff --
Please add the version to this annotation
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by swill <gi...@git.apache.org>.
Github user swill commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-214889780
@rhtyd ya, I am not too worried about this PR. I think we are all set with this PR once @jburwell has a chance to do a quick once through. This PR just prompted me to ask the question more generally. This is probably not the right place for this conversation, but I wanted to see if there was an approach we should be taking going forward.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by wido <gi...@git.apache.org>.
Github user wido commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r60825823
--- Diff: api/src/org/apache/cloudstack/api/command/admin/outofbandmanagement/ConfigureOutOfBandManagementCmd.java ---
@@ -0,0 +1,127 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.api.command.admin.outofbandmanagement;
+
+import com.cloud.exception.ConcurrentOperationException;
+import com.cloud.exception.InsufficientCapacityException;
+import com.cloud.exception.NetworkRuleConflictException;
+import com.cloud.exception.ResourceAllocationException;
+import com.cloud.exception.ResourceUnavailableException;
+import com.cloud.host.Host;
+import com.google.common.base.Strings;
+import com.google.common.collect.ImmutableMap;
+import org.apache.cloudstack.acl.RoleType;
+import org.apache.cloudstack.api.APICommand;
+import org.apache.cloudstack.api.ApiConstants;
+import org.apache.cloudstack.api.ApiErrorCode;
+import org.apache.cloudstack.api.BaseCmd;
+import org.apache.cloudstack.api.Parameter;
+import org.apache.cloudstack.api.ServerApiException;
+import org.apache.cloudstack.api.response.HostResponse;
+import org.apache.cloudstack.api.response.OutOfBandManagementResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagement;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagementService;
+
+import javax.inject.Inject;
+
+@APICommand(name = "configureOutOfBandManagement", description = "Configures a host's out-of-band management interface",
+ responseObject = OutOfBandManagementResponse.class, requestHasSensitiveInfo = true, responseHasSensitiveInfo = false, authorized = {RoleType.Admin})
+public class ConfigureOutOfBandManagementCmd extends BaseCmd {
+ @Inject
+ private OutOfBandManagementService outOfBandManagementService;
+
+ /////////////////////////////////////////////////////
+ //////////////// API parameters /////////////////////
+ /////////////////////////////////////////////////////
+
+ @Parameter(name = ApiConstants.HOST_ID, type = CommandType.UUID, entityType = HostResponse.class, required = true, description = "the ID of the host")
+ private Long hostId;
+
+ @Parameter(name = ApiConstants.DRIVER, type = CommandType.STRING, required = true, description = "the host management interface driver, for example: ipmitool")
+ private String driver;
+
+ @Parameter(name = ApiConstants.IP_ADDRESS, type = CommandType.STRING, required = true, description = "the host management interface IP address")
+ private String address;
+
+ @Parameter(name = ApiConstants.PORT, type = CommandType.STRING, required = true, description = "the host management interface port")
--- End diff --
Why is the port numer a String and not a Integer?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by DaanHoogland <gi...@git.apache.org>.
Github user DaanHoogland commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r60725627
--- Diff: api/src/org/apache/cloudstack/api/command/admin/outofbandmanagement/DisableOutOfBandManagementForHostCmd.java ---
@@ -0,0 +1,107 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.api.command.admin.outofbandmanagement;
+
+import com.cloud.event.EventTypes;
+import com.cloud.exception.ConcurrentOperationException;
+import com.cloud.exception.InsufficientCapacityException;
+import com.cloud.exception.NetworkRuleConflictException;
+import com.cloud.exception.ResourceAllocationException;
+import com.cloud.exception.ResourceUnavailableException;
+import com.cloud.host.Host;
+import org.apache.cloudstack.acl.RoleType;
+import org.apache.cloudstack.api.APICommand;
+import org.apache.cloudstack.api.ApiConstants;
+import org.apache.cloudstack.api.ApiErrorCode;
+import org.apache.cloudstack.api.BaseAsyncCmd;
+import org.apache.cloudstack.api.BaseCmd;
+import org.apache.cloudstack.api.Parameter;
+import org.apache.cloudstack.api.ServerApiException;
+import org.apache.cloudstack.api.response.HostResponse;
+import org.apache.cloudstack.api.response.OutOfBandManagementResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagementService;
+
+import javax.inject.Inject;
+
+@APICommand(name = "disableOutOfBandManagementForHost", description = "Disables out-of-band management for a host",
+ responseObject = OutOfBandManagementResponse.class, requestHasSensitiveInfo = false, responseHasSensitiveInfo = false, authorized = {RoleType.Admin})
+public class DisableOutOfBandManagementForHostCmd extends BaseAsyncCmd {
+
+ @Inject
+ private OutOfBandManagementService outOfBandManagementService;
+
+ /////////////////////////////////////////////////////
+ //////////////// API parameters /////////////////////
+ /////////////////////////////////////////////////////
+
+ @Parameter(name = ApiConstants.HOST_ID, type = BaseCmd.CommandType.UUID, required = true, entityType = HostResponse.class, description = "the ID of the host")
+ private Long hostId;
+
+ /////////////////////////////////////////////////////
+ /////////////// API Implementation///////////////////
+ /////////////////////////////////////////////////////
+
+ private void validateParams() {
--- End diff --
pity your new Annotation parameter for @Parameter, validators is not in yet.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61847460
--- Diff: server/test/com/cloud/resource/MockResourceManagerImpl.java ---
@@ -172,6 +173,12 @@ public Cluster getCluster(final Long clusterId) {
return null;
}
+ @Override
+ public DataCenter getZone(Long zoneId) {
+ // TODO Auto-generated method stub
--- End diff --
This is a test mock, from test code
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61850479
--- Diff: server/test/org/apache/cloudstack/outofbandmanagement/OutOfBandManagementServiceTest.java ---
@@ -0,0 +1,119 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.outofbandmanagement;
+
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.google.common.collect.ImmutableMap;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverResponse;
+import org.junit.Assert;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.mockito.runners.MockitoJUnitRunner;
+
+@RunWith(MockitoJUnitRunner.class)
+public class OutOfBandManagementServiceTest {
+
+ OutOfBandManagementServiceImpl oobmService = new OutOfBandManagementServiceImpl();
+
+ @Test
+ public void testOutOfBandManagementDriverResponseEvent() {
+ OutOfBandManagementDriverResponse r = new OutOfBandManagementDriverResponse("some result", "some error", false);
+
+ r.setSuccess(false);
+ r.setAuthFailure(false);
+ Assert.assertEquals(r.toEvent(), OutOfBandManagement.PowerState.Event.Unknown);
+
+ r.setSuccess(false);
+ r.setAuthFailure(true);
+ Assert.assertEquals(r.toEvent(), OutOfBandManagement.PowerState.Event.AuthError);
+
+ r.setAuthFailure(false);
+ r.setSuccess(true);
+ r.setPowerState(OutOfBandManagement.PowerState.On);
+ Assert.assertEquals(r.toEvent(), OutOfBandManagement.PowerState.Event.On);
+
+ r.setPowerState(OutOfBandManagement.PowerState.Off);
+ Assert.assertEquals(r.toEvent(), OutOfBandManagement.PowerState.Event.Off);
+
+ r.setPowerState(OutOfBandManagement.PowerState.Disabled);
+ Assert.assertEquals(r.toEvent(), OutOfBandManagement.PowerState.Event.Disabled);
+ }
+
+ private ImmutableMap<OutOfBandManagement.Option, String> buildRandomOptionsMap() {
+ ImmutableMap.Builder<OutOfBandManagement.Option, String> builder = new ImmutableMap.Builder<>();
+ builder.put(OutOfBandManagement.Option.ADDRESS, "localhost");
+ builder.put(OutOfBandManagement.Option.DRIVER, "ipmitool");
+ return builder.build();
+ }
+
+ @Test
+ public void testUpdateOutOfBandManagementConfigValid() {
+ OutOfBandManagement config = new OutOfBandManagementVO(123L);
+ Assert.assertEquals(config.getPowerState(), OutOfBandManagement.PowerState.Disabled);
+ config = oobmService.updateConfig(config, buildRandomOptionsMap());
+ Assert.assertEquals(config.getAddress(), "localhost");
+ Assert.assertEquals(config.getDriver(), "ipmitool");
+ Assert.assertEquals(config.getPowerState(), OutOfBandManagement.PowerState.Disabled);
+ }
+
+ @Test
+ public void testUpdateOutOfBandManagementConfigInValid() {
+ try {
+ oobmService.updateConfig(null, buildRandomOptionsMap());
+ Assert.fail("CloudRuntimeException was expect for out-of-band management not configured for the host");
+ } catch (CloudRuntimeException ignored) {
+ }
+
+ try {
+ oobmService.updateConfig(null, null);
+ Assert.fail("CloudRuntimeException was expect for out-of-band management not configured for the host");
+ } catch (CloudRuntimeException ignored) {
+ }
+
+ OutOfBandManagement config = new OutOfBandManagementVO(123L);
+ config.setAddress(null);
+ config = oobmService.updateConfig(config, null);
+ Assert.assertEquals(config.getAddress(), null);
+ Assert.assertEquals(config.getPowerState(), OutOfBandManagement.PowerState.Disabled);
--- End diff --
Fixed, split into three unit tests
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61813461
--- Diff: server/src/org/apache/cloudstack/outofbandmanagement/OutOfBandManagementServiceImpl.java ---
@@ -0,0 +1,532 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.outofbandmanagement;
+
+import com.cloud.alert.AlertManager;
+import com.cloud.dc.ClusterDetailsDao;
+import com.cloud.dc.ClusterDetailsVO;
+import com.cloud.dc.DataCenter;
+import com.cloud.dc.DataCenterDetailVO;
+import com.cloud.dc.dao.DataCenterDetailsDao;
+import com.cloud.domain.Domain;
+import com.cloud.event.ActionEvent;
+import com.cloud.event.ActionEventUtils;
+import com.cloud.event.EventTypes;
+import com.cloud.host.Host;
+import com.cloud.host.dao.HostDao;
+import com.cloud.org.Cluster;
+import com.cloud.utils.component.Manager;
+import com.cloud.utils.component.ManagerBase;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.db.Transaction;
+import com.cloud.utils.db.TransactionCallback;
+import com.cloud.utils.db.TransactionStatus;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.fsm.NoTransitionException;
+import com.google.common.base.Strings;
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import com.google.common.collect.ImmutableMap;
+import org.apache.cloudstack.api.response.OutOfBandManagementResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.framework.config.ConfigKey;
+import org.apache.cloudstack.framework.config.Configurable;
+import org.apache.cloudstack.outofbandmanagement.dao.OutOfBandManagementDao;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverChangePasswordCommand;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverPowerCommand;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverResponse;
+import org.apache.cloudstack.utils.identity.ManagementServerNode;
+import org.apache.log4j.Logger;
+import org.springframework.stereotype.Component;
+
+import javax.ejb.Local;
+import javax.inject.Inject;
+import javax.naming.ConfigurationException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+@Component
+@Local(value = {OutOfBandManagementService.class})
+public class OutOfBandManagementServiceImpl extends ManagerBase implements OutOfBandManagementService, Manager, Configurable {
+ public static final Logger LOG = Logger.getLogger(OutOfBandManagementServiceImpl.class);
+
+ @Inject
+ private ClusterDetailsDao clusterDetailsDao;
+ @Inject
+ private DataCenterDetailsDao dataCenterDetailsDao;
+ @Inject
+ private OutOfBandManagementDao outOfBandManagementDao;
+ @Inject
+ private HostDao hostDao;
+ @Inject
+ private AlertManager alertMgr;
+
+ private String name;
+ private long serviceId;
+
+ private List<OutOfBandManagementDriver> outOfBandManagementDrivers = new ArrayList<>();
+ private Map<String, OutOfBandManagementDriver> outOfBandManagementDriversMap = new HashMap<String, OutOfBandManagementDriver>();
+
+ private static final String OOBM_ENABLED_DETAIL = "outOfBandManagementEnabled";
+ private static final int ACQUIRE_GLOBAL_LOCK_TIMEOUT_FOR_HOST = 120;
+
+ private Cache<Long, Long> hostAlertCache;
+ private static ExecutorService backgroundSyncExecutor;
+
+ private String getOutOfBandManagementHostLock(long id) {
+ return "oobm.host." + id;
+ }
+
+ private void initializeDriversMap() {
+ if (outOfBandManagementDriversMap.isEmpty() && outOfBandManagementDrivers != null && outOfBandManagementDrivers.size() > 0) {
+ for (final OutOfBandManagementDriver driver : outOfBandManagementDrivers) {
+ outOfBandManagementDriversMap.put(driver.getName().toLowerCase(), driver);
+ }
+ LOG.debug("Discovered out-of-band management drivers configured in the OutOfBandManagementService");
+ }
+ }
+
+ private OutOfBandManagementDriver getDriver(final OutOfBandManagement outOfBandManagementConfig) {
+ if (!Strings.isNullOrEmpty(outOfBandManagementConfig.getDriver())) {
+ final OutOfBandManagementDriver driver = outOfBandManagementDriversMap.get(outOfBandManagementConfig.getDriver());
+ if (driver != null) {
+ return driver;
+ }
+ }
+ throw new CloudRuntimeException("Configured out-of-band management driver is not available. Aborting any out-of-band management action.");
+ }
+
+ protected OutOfBandManagement updateConfig(final OutOfBandManagement outOfBandManagementConfig, final ImmutableMap<OutOfBandManagement.Option, String> options) {
+ if (outOfBandManagementConfig == null) {
+ throw new CloudRuntimeException("Out-of-band management is not configured for the host. Aborting.");
+ }
+ if (options == null) {
+ return outOfBandManagementConfig;
+ }
+ for (OutOfBandManagement.Option option: options.keySet()) {
+ final String value = options.get(option);
+ if (Strings.isNullOrEmpty(value)) {
+ continue;
+ }
+ switch (option) {
+ case DRIVER:
+ outOfBandManagementConfig.setDriver(value);
+ break;
+ case ADDRESS:
+ outOfBandManagementConfig.setAddress(value);
+ break;
+ case PORT:
+ outOfBandManagementConfig.setPort(Integer.parseInt(value));
+ break;
+ case USERNAME:
+ outOfBandManagementConfig.setUsername(value);
+ break;
+ case PASSWORD:
+ outOfBandManagementConfig.setPassword(value);
+ break;
+ }
+ }
+ return outOfBandManagementConfig;
+ }
+
+ protected ImmutableMap<OutOfBandManagement.Option, String> getOptions(final OutOfBandManagement outOfBandManagementConfig) {
+ final ImmutableMap.Builder<OutOfBandManagement.Option, String> optionsBuilder = ImmutableMap.builder();
+ if (outOfBandManagementConfig == null) {
+ throw new CloudRuntimeException("Out-of-band management is not configured for the host. Aborting.");
+ }
+ for (OutOfBandManagement.Option option: OutOfBandManagement.Option.values()) {
+ String value = null;
+ switch (option) {
+ case DRIVER:
+ value = outOfBandManagementConfig.getDriver();
+ break;
+ case ADDRESS:
+ value = outOfBandManagementConfig.getAddress();
+ break;
+ case PORT:
+ if (outOfBandManagementConfig.getPort() != null) {
+ value = String.valueOf(outOfBandManagementConfig.getPort());
+ }
+ break;
+ case USERNAME:
+ value = outOfBandManagementConfig.getUsername();
+ break;
+ case PASSWORD:
+ value = outOfBandManagementConfig.getPassword();
+ break;
+ }
+ if (value != null) {
+ optionsBuilder.put(option, value);
+ }
+ }
+ return optionsBuilder.build();
+ }
+
+ private void sendAuthError(final Host host, final String message) {
+ try {
+ hostAlertCache.asMap().putIfAbsent(host.getId(), 0L);
+ Long sentCount = hostAlertCache.asMap().get(host.getId());
+ if (sentCount != null && sentCount <= 0) {
+ boolean concurrentUpdateResult = hostAlertCache.asMap().replace(host.getId(), sentCount, sentCount+1L);
+ if (concurrentUpdateResult) {
+ final String subject = String.format("Out-of-band management auth-error detected for host:%d in cluster:%d, zone:%d", host.getId(), host.getClusterId(), host.getDataCenterId());
+ LOG.error(subject + ": " + message);
+ alertMgr.sendAlert(AlertManager.AlertType.ALERT_TYPE_OOBM_AUTH_ERROR, host.getDataCenterId(), host.getPodId(), subject, message);
+ }
+ }
+ } catch (Exception ignored) {
+ }
+ }
+
+ private boolean transitionPowerState(OutOfBandManagement.PowerState.Event event, OutOfBandManagement outOfBandManagementHost) {
+ if (outOfBandManagementHost == null) {
+ return false;
+ }
+ OutOfBandManagement.PowerState currentPowerState = outOfBandManagementHost.getPowerState();
+ try {
+ OutOfBandManagement.PowerState newPowerState = OutOfBandManagement.PowerState.getStateMachine().getNextState(currentPowerState, event);
+ boolean result = outOfBandManagementDao.updateState(currentPowerState, event, newPowerState, outOfBandManagementHost, null);
+ if (result) {
+ final String message = String.format("Transitioned out-of-band management power state from:%s to:%s due to event:%s for the host id:%d", currentPowerState, newPowerState, event, outOfBandManagementHost.getHostId());
+ LOG.debug(message);
+ ActionEventUtils.onActionEvent(CallContext.current().getCallingUserId(), CallContext.current().getCallingAccountId(), Domain.ROOT_DOMAIN,
+ EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_POWERSTATE_TRANSITION, message);
+ }
+ return result;
+ } catch (NoTransitionException ignored) {
+ LOG.trace(String.format("Unable to transition out-of-band management power state for host id=%s for the event=%s and current power state=%s", outOfBandManagementHost.getHostId(), event, currentPowerState));
+ }
+ return false;
+ }
+
+ private boolean isOutOfBandManagementEnabledForZone(Long zoneId) {
+ if (zoneId == null) {
+ return true;
+ }
+ final DataCenterDetailVO zoneDetails = dataCenterDetailsDao.findDetail(zoneId, OOBM_ENABLED_DETAIL);
+ if (zoneDetails != null && !Strings.isNullOrEmpty(zoneDetails.getValue()) && !Boolean.valueOf(zoneDetails.getValue())) {
+ return false;
+ }
+ return true;
+ }
+
+ private boolean isOutOfBandManagementEnabledForCluster(Long clusterId) {
+ if (clusterId == null) {
+ return true;
+ }
+ final ClusterDetailsVO clusterDetails = clusterDetailsDao.findDetail(clusterId, OOBM_ENABLED_DETAIL);
+ if (clusterDetails != null && !Strings.isNullOrEmpty(clusterDetails.getValue()) && !Boolean.valueOf(clusterDetails.getValue())) {
+ return false;
+ }
+ return true;
+ }
+
+ private boolean isOutOfBandManagementEnabledForHost(Long hostId) {
+ if (hostId == null) {
+ return false;
+ }
+ final OutOfBandManagement outOfBandManagementConfig = outOfBandManagementDao.findByHost(hostId);
+ if (outOfBandManagementConfig == null || !outOfBandManagementConfig.isEnabled()) {
+ return false;
+ }
+ return true;
+ }
+
+ private void checkOutOfBandManagementEnabledByZoneClusterHost(final Host host) {
+ if (!isOutOfBandManagementEnabledForZone(host.getDataCenterId())) {
+ throw new CloudRuntimeException("Out-of-band management is disabled for the host's zone. Aborting Operation.");
+ }
+ if (!isOutOfBandManagementEnabledForCluster(host.getClusterId())) {
+ throw new CloudRuntimeException("Out-of-band management is disabled for the host's cluster. Aborting Operation.");
+ }
+ if (!isOutOfBandManagementEnabledForHost(host.getId())) {
+ throw new CloudRuntimeException("Out-of-band management is disabled or not configured for the host. Aborting Operation.");
+ }
+ }
+
+ public boolean isOutOfBandManagementEnabled(final Host host) {
+ return isOutOfBandManagementEnabledForZone(host.getDataCenterId())
+ && isOutOfBandManagementEnabledForCluster(host.getClusterId())
+ && isOutOfBandManagementEnabledForHost(host.getId());
+ }
+
+ public boolean transitionPowerStateToDisabled(List<? extends Host> hosts) {
+ boolean result = true;
+ for (Host host : hosts) {
+ result = result && transitionPowerState(OutOfBandManagement.PowerState.Event.Disabled,
+ outOfBandManagementDao.findByHost(host.getId()));
+ }
+ return result;
+ }
+
+ public void submitBackgroundPowerSyncTask(final Host host) {
+ if (host != null) {
+ // Note: This is a blocking queue based executor
+ backgroundSyncExecutor.submit(new OutOfBandManagementBackgroundTask(this, host, OutOfBandManagement.PowerOperation.STATUS));
+ }
+ }
+
+ private OutOfBandManagementResponse buildEnableDisableResponse(final boolean enabled) {
+ final OutOfBandManagementResponse response = new OutOfBandManagementResponse();
+ response.setEnabled(enabled);
+ response.setSuccess(true);
+ return response;
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_ENABLEDISABLE, eventDescription = "enabling/disabling out-of-band management on a zone")
+ public OutOfBandManagementResponse enableDisableOutOfBandManagement(final DataCenter zone, final boolean enabled) {
+ dataCenterDetailsDao.persist(zone.getId(), OOBM_ENABLED_DETAIL, String.valueOf(enabled));
+ if (!enabled) {
+ transitionPowerStateToDisabled(hostDao.findByDataCenterId(zone.getId()));
+ }
+ return buildEnableDisableResponse(enabled);
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_ENABLEDISABLE, eventDescription = "enabling/disabling out-of-band management on a cluster")
+ public OutOfBandManagementResponse enableDisableOutOfBandManagement(final Cluster cluster, final boolean enabled) {
+ clusterDetailsDao.persist(cluster.getId(), OOBM_ENABLED_DETAIL, String.valueOf(enabled));
+ if (!enabled) {
+ transitionPowerStateToDisabled(hostDao.findByClusterId(cluster.getId()));
+ }
+ return buildEnableDisableResponse(enabled);
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_ENABLEDISABLE, eventDescription = "enabling/disabling out-of-band management on a host")
+ public OutOfBandManagementResponse enableDisableOutOfBandManagement(final Host host, final boolean enabled) {
+ final OutOfBandManagement outOfBandManagementConfig = outOfBandManagementDao.findByHost(host.getId());
+ if (outOfBandManagementConfig == null) {
+ final OutOfBandManagementResponse response = new OutOfBandManagementResponse(null);
+ response.setSuccess(false);
+ response.setResultDescription("Out-of-band management is not configured for the host. Please configure the host before enabling/disabling it.");
+ return response;
+ }
+ hostAlertCache.invalidate(host.getId());
+ outOfBandManagementConfig.setEnabled(enabled);
+ boolean updateResult = outOfBandManagementDao.update(outOfBandManagementConfig.getId(), (OutOfBandManagementVO) outOfBandManagementConfig);
+ if (updateResult && !enabled) {
+ transitionPowerStateToDisabled(Collections.singletonList(host));
+ }
+ return buildEnableDisableResponse(enabled && updateResult);
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_CONFIGURE, eventDescription = "updating out-of-band management configuration")
+ public OutOfBandManagementResponse configureOutOfBandManagement(final Host host, final ImmutableMap<OutOfBandManagement.Option, String> options) {
+ OutOfBandManagement outOfBandManagementConfig = outOfBandManagementDao.findByHost(host.getId());
+ if (outOfBandManagementConfig == null) {
+ outOfBandManagementConfig = outOfBandManagementDao.persist(new OutOfBandManagementVO(host.getId()));
+ }
+ outOfBandManagementConfig = updateConfig(outOfBandManagementConfig, options);
+ if (Strings.isNullOrEmpty(outOfBandManagementConfig.getDriver()) || !outOfBandManagementDriversMap.containsKey(outOfBandManagementConfig.getDriver().toLowerCase())) {
+ throw new CloudRuntimeException("Out-of-band management driver is not available. Please provide a valid driver name.");
+ }
+
+ boolean updatedConfig = outOfBandManagementDao.update(outOfBandManagementConfig.getId(), (OutOfBandManagementVO) outOfBandManagementConfig);
+ CallContext.current().setEventDetails("host id:" + host.getId() + " configuration:" + outOfBandManagementConfig.getAddress() + ":" + outOfBandManagementConfig.getPort());
+
+ if (!updatedConfig) {
+ throw new CloudRuntimeException("Failed to update out-of-band management config for the host in the database.");
+ }
+
+ String result = "Out-of-band management successfully configured for the host";
+ LOG.debug(result);
+
+ final OutOfBandManagementResponse response = new OutOfBandManagementResponse(outOfBandManagementDao.findByHost(host.getId()));
+ response.setResultDescription(result);
+ response.setSuccess(true);
+ return response;
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_ACTION, eventDescription = "issuing Host out-of-band management action", async = true)
+ public OutOfBandManagementResponse executeOutOfBandManagementPowerOperation(final Host host, final OutOfBandManagement.PowerOperation powerOperation, final Long timeout) {
+ checkOutOfBandManagementEnabledByZoneClusterHost(host);
+ final OutOfBandManagement outOfBandManagementConfig = outOfBandManagementDao.findByHost(host.getId());
+ final ImmutableMap<OutOfBandManagement.Option, String> options = getOptions(outOfBandManagementConfig);
+ final OutOfBandManagementDriver driver = getDriver(outOfBandManagementConfig);
+
+ Long actionTimeOut = timeout;
+ if (actionTimeOut == null) {
+ actionTimeOut = OutOfBandManagementActionTimeout.valueIn(host.getClusterId());
+ }
+
+ final OutOfBandManagementDriverPowerCommand cmd = new OutOfBandManagementDriverPowerCommand(options, actionTimeOut, powerOperation);
+ final OutOfBandManagementDriverResponse driverResponse = driver.execute(cmd);
+
+ if (driverResponse == null) {
+ throw new CloudRuntimeException(String.format("Out-of-band Management action (%s) on host (%s) failed due to no response from the driver", powerOperation, host.getUuid()));
+ }
+
+ if (powerOperation.equals(OutOfBandManagement.PowerOperation.STATUS)) {
+ transitionPowerState(driverResponse.toEvent(), outOfBandManagementConfig);
+ }
+
+ if (!driverResponse.isSuccess()) {
+ String errorMessage = String.format("Out-of-band Management action (%s) on host (%s) failed with error: %s", powerOperation, host.getUuid(), driverResponse.getError());
+ if (driverResponse.hasAuthFailure()) {
+ errorMessage = String.format("Out-of-band Management action (%s) on host (%s) failed due to authentication error: %s. Please check configured credentials.", powerOperation, host.getUuid(), driverResponse.getError());
+ sendAuthError(host, errorMessage);
+ }
+ if (!powerOperation.equals(OutOfBandManagement.PowerOperation.STATUS)) {
+ LOG.debug(errorMessage);
+ }
+ throw new CloudRuntimeException(errorMessage);
+ }
+
+ final OutOfBandManagementResponse response = new OutOfBandManagementResponse(outOfBandManagementDao.findByHost(host.getId()));
+ response.setSuccess(driverResponse.isSuccess());
+ response.setResultDescription(driverResponse.getResult());
+ response.setId(host.getUuid());
+ response.setOutOfBandManagementAction(powerOperation.toString());
+ return response;
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_CHANGE_PASSWORD, eventDescription = "updating out-of-band management password")
+ public OutOfBandManagementResponse changeOutOfBandManagementPassword(final Host host, final String newPassword) {
+ checkOutOfBandManagementEnabledByZoneClusterHost(host);
+ if (Strings.isNullOrEmpty(newPassword)) {
+ throw new CloudRuntimeException(String.format("Cannot change out-of-band management password as provided new-password is null or empty for the host %s.", host.getUuid()));
+ }
+ GlobalLock outOfBandManagementHostLock = GlobalLock.getInternLock(getOutOfBandManagementHostLock(host.getId()));
+ try {
+ if (outOfBandManagementHostLock.lock(ACQUIRE_GLOBAL_LOCK_TIMEOUT_FOR_HOST)) {
+ try {
+ final OutOfBandManagement outOfBandManagementConfig = outOfBandManagementDao.findByHost(host.getId());
+
+ final ImmutableMap<OutOfBandManagement.Option, String> options = getOptions(outOfBandManagementConfig);
+ if (!(options.containsKey(OutOfBandManagement.Option.PASSWORD) && !Strings.isNullOrEmpty(options.get(OutOfBandManagement.Option.PASSWORD)))) {
+ throw new CloudRuntimeException(String.format("Cannot change out-of-band management password as we've no previously configured password for the host %s.", host.getUuid()));
+ }
+ final OutOfBandManagementDriver driver = getDriver(outOfBandManagementConfig);
+
+ final OutOfBandManagementDriverChangePasswordCommand cmd = new OutOfBandManagementDriverChangePasswordCommand(options, OutOfBandManagementActionTimeout.valueIn(host.getClusterId()), newPassword);
+ final OutOfBandManagementDriverResponse driverResponse;
+ try {
+ driverResponse = driver.execute(cmd);
+ } catch (Exception e) {
+ LOG.error("Out-of-band management change password failed due to driver error: " + e.getMessage());
+ throw new CloudRuntimeException(String.format("Failed to change out-of-band management password for host (%s) due to driver error: %s", host.getUuid(), e.getMessage()));
+ }
+
+ if (!driverResponse.isSuccess()) {
+ throw new CloudRuntimeException(String.format("Failed to change out-of-band management password for host (%s) with error: %s", host.getUuid(), driverResponse.getError()));
+ }
+
+ final boolean updatedConfigResult = Transaction.execute(new TransactionCallback<Boolean>() {
+ @Override
+ public Boolean doInTransaction(TransactionStatus status) {
+ OutOfBandManagement updatedOutOfBandManagementConfig = outOfBandManagementDao.findByHost(host.getId());
+ updatedOutOfBandManagementConfig.setPassword(newPassword);
+ return outOfBandManagementDao.update(updatedOutOfBandManagementConfig.getId(), (OutOfBandManagementVO) updatedOutOfBandManagementConfig);
+ }
+ });
+
+ if (!updatedConfigResult) {
+ LOG.error(String.format("Succeeded to change out-of-band management password but failed to updated in database the new password:%s for the host id:%d", newPassword, host.getId()));
+ }
+
+ final OutOfBandManagementResponse response = new OutOfBandManagementResponse();
+ response.setSuccess(updatedConfigResult && driverResponse.isSuccess());
+ response.setResultDescription(driverResponse.getResult());
+ response.setId(host.getUuid());
+ return response;
+ } finally {
+ outOfBandManagementHostLock.unlock();
+ }
+ } else {
+ LOG.error("Unable to acquire synchronization lock to change out-of-band management password for host id: " + host.getId());
+ throw new CloudRuntimeException(String.format("Unable to acquire lock to change out-of-band management password for host (%s), please try after some time.", host.getUuid()));
+ }
+ } finally {
+ outOfBandManagementHostLock.releaseRef();
+ }
+ }
+
+ @Override
+ public String getName() {
+ return name;
+ }
+
+ @Override
+ public long getId() {
+ return serviceId;
+ }
+
+ @Override
+ public boolean configure(final String name, final Map<String, Object> params) throws ConfigurationException {
+ this.name = name;
+ this.serviceId = ManagementServerNode.getManagementServerId();
+
+ final int poolSize = OutOfBandManagementSyncThreadPoolSize.value();
+
+ hostAlertCache = CacheBuilder.newBuilder()
+ .concurrencyLevel(4)
+ .weakKeys()
+ .maximumSize(100 * poolSize)
+ .expireAfterWrite(1, TimeUnit.DAYS)
+ .build();
+
+ backgroundSyncExecutor = new ThreadPoolExecutor(poolSize, poolSize,
+ 0L, TimeUnit.MILLISECONDS,
+ new ArrayBlockingQueue<Runnable>(10 * poolSize, true), new ThreadPoolExecutor.CallerRunsPolicy());
+
+ LOG.info("Starting out-of-band management background sync executor with thread pool-size=" + poolSize + " and background sync thread interval=" + OutOfBandManagementSyncThreadInterval.value() + "s");
+ return true;
+ }
+
+ @Override
+ public boolean start() {
+ initializeDriversMap();
+ return true;
+ }
+
+ @Override
+ public boolean stop() {
+ backgroundSyncExecutor.shutdown();
+ outOfBandManagementDao.expireOutOfBandManagementOwnershipByServer(getId());
--- End diff --
How will host handoff occur if the management server loses its database connection?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61807854
--- Diff: server/src/org/apache/cloudstack/outofbandmanagement/OutOfBandManagementServiceImpl.java ---
@@ -0,0 +1,532 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.outofbandmanagement;
+
+import com.cloud.alert.AlertManager;
+import com.cloud.dc.ClusterDetailsDao;
+import com.cloud.dc.ClusterDetailsVO;
+import com.cloud.dc.DataCenter;
+import com.cloud.dc.DataCenterDetailVO;
+import com.cloud.dc.dao.DataCenterDetailsDao;
+import com.cloud.domain.Domain;
+import com.cloud.event.ActionEvent;
+import com.cloud.event.ActionEventUtils;
+import com.cloud.event.EventTypes;
+import com.cloud.host.Host;
+import com.cloud.host.dao.HostDao;
+import com.cloud.org.Cluster;
+import com.cloud.utils.component.Manager;
+import com.cloud.utils.component.ManagerBase;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.db.Transaction;
+import com.cloud.utils.db.TransactionCallback;
+import com.cloud.utils.db.TransactionStatus;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.fsm.NoTransitionException;
+import com.google.common.base.Strings;
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import com.google.common.collect.ImmutableMap;
+import org.apache.cloudstack.api.response.OutOfBandManagementResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.framework.config.ConfigKey;
+import org.apache.cloudstack.framework.config.Configurable;
+import org.apache.cloudstack.outofbandmanagement.dao.OutOfBandManagementDao;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverChangePasswordCommand;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverPowerCommand;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverResponse;
+import org.apache.cloudstack.utils.identity.ManagementServerNode;
+import org.apache.log4j.Logger;
+import org.springframework.stereotype.Component;
+
+import javax.ejb.Local;
+import javax.inject.Inject;
+import javax.naming.ConfigurationException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+@Component
+@Local(value = {OutOfBandManagementService.class})
+public class OutOfBandManagementServiceImpl extends ManagerBase implements OutOfBandManagementService, Manager, Configurable {
+ public static final Logger LOG = Logger.getLogger(OutOfBandManagementServiceImpl.class);
+
+ @Inject
+ private ClusterDetailsDao clusterDetailsDao;
+ @Inject
+ private DataCenterDetailsDao dataCenterDetailsDao;
+ @Inject
+ private OutOfBandManagementDao outOfBandManagementDao;
+ @Inject
+ private HostDao hostDao;
+ @Inject
+ private AlertManager alertMgr;
+
+ private String name;
+ private long serviceId;
+
+ private List<OutOfBandManagementDriver> outOfBandManagementDrivers = new ArrayList<>();
+ private Map<String, OutOfBandManagementDriver> outOfBandManagementDriversMap = new HashMap<String, OutOfBandManagementDriver>();
+
+ private static final String OOBM_ENABLED_DETAIL = "outOfBandManagementEnabled";
+ private static final int ACQUIRE_GLOBAL_LOCK_TIMEOUT_FOR_HOST = 120;
+
+ private Cache<Long, Long> hostAlertCache;
+ private static ExecutorService backgroundSyncExecutor;
+
+ private String getOutOfBandManagementHostLock(long id) {
+ return "oobm.host." + id;
+ }
+
+ private void initializeDriversMap() {
+ if (outOfBandManagementDriversMap.isEmpty() && outOfBandManagementDrivers != null && outOfBandManagementDrivers.size() > 0) {
+ for (final OutOfBandManagementDriver driver : outOfBandManagementDrivers) {
+ outOfBandManagementDriversMap.put(driver.getName().toLowerCase(), driver);
+ }
+ LOG.debug("Discovered out-of-band management drivers configured in the OutOfBandManagementService");
+ }
+ }
+
+ private OutOfBandManagementDriver getDriver(final OutOfBandManagement outOfBandManagementConfig) {
+ if (!Strings.isNullOrEmpty(outOfBandManagementConfig.getDriver())) {
+ final OutOfBandManagementDriver driver = outOfBandManagementDriversMap.get(outOfBandManagementConfig.getDriver());
+ if (driver != null) {
+ return driver;
+ }
+ }
+ throw new CloudRuntimeException("Configured out-of-band management driver is not available. Aborting any out-of-band management action.");
+ }
+
+ protected OutOfBandManagement updateConfig(final OutOfBandManagement outOfBandManagementConfig, final ImmutableMap<OutOfBandManagement.Option, String> options) {
+ if (outOfBandManagementConfig == null) {
+ throw new CloudRuntimeException("Out-of-band management is not configured for the host. Aborting.");
+ }
+ if (options == null) {
+ return outOfBandManagementConfig;
+ }
+ for (OutOfBandManagement.Option option: options.keySet()) {
+ final String value = options.get(option);
+ if (Strings.isNullOrEmpty(value)) {
+ continue;
+ }
+ switch (option) {
+ case DRIVER:
+ outOfBandManagementConfig.setDriver(value);
+ break;
+ case ADDRESS:
+ outOfBandManagementConfig.setAddress(value);
+ break;
+ case PORT:
+ outOfBandManagementConfig.setPort(Integer.parseInt(value));
+ break;
+ case USERNAME:
+ outOfBandManagementConfig.setUsername(value);
+ break;
+ case PASSWORD:
+ outOfBandManagementConfig.setPassword(value);
+ break;
+ }
+ }
+ return outOfBandManagementConfig;
+ }
+
+ protected ImmutableMap<OutOfBandManagement.Option, String> getOptions(final OutOfBandManagement outOfBandManagementConfig) {
+ final ImmutableMap.Builder<OutOfBandManagement.Option, String> optionsBuilder = ImmutableMap.builder();
+ if (outOfBandManagementConfig == null) {
+ throw new CloudRuntimeException("Out-of-band management is not configured for the host. Aborting.");
+ }
+ for (OutOfBandManagement.Option option: OutOfBandManagement.Option.values()) {
+ String value = null;
+ switch (option) {
+ case DRIVER:
+ value = outOfBandManagementConfig.getDriver();
+ break;
+ case ADDRESS:
+ value = outOfBandManagementConfig.getAddress();
+ break;
+ case PORT:
+ if (outOfBandManagementConfig.getPort() != null) {
+ value = String.valueOf(outOfBandManagementConfig.getPort());
+ }
+ break;
+ case USERNAME:
+ value = outOfBandManagementConfig.getUsername();
+ break;
+ case PASSWORD:
+ value = outOfBandManagementConfig.getPassword();
+ break;
+ }
+ if (value != null) {
+ optionsBuilder.put(option, value);
+ }
+ }
+ return optionsBuilder.build();
+ }
+
+ private void sendAuthError(final Host host, final String message) {
+ try {
+ hostAlertCache.asMap().putIfAbsent(host.getId(), 0L);
+ Long sentCount = hostAlertCache.asMap().get(host.getId());
+ if (sentCount != null && sentCount <= 0) {
+ boolean concurrentUpdateResult = hostAlertCache.asMap().replace(host.getId(), sentCount, sentCount+1L);
+ if (concurrentUpdateResult) {
+ final String subject = String.format("Out-of-band management auth-error detected for host:%d in cluster:%d, zone:%d", host.getId(), host.getClusterId(), host.getDataCenterId());
+ LOG.error(subject + ": " + message);
+ alertMgr.sendAlert(AlertManager.AlertType.ALERT_TYPE_OOBM_AUTH_ERROR, host.getDataCenterId(), host.getPodId(), subject, message);
+ }
+ }
+ } catch (Exception ignored) {
+ }
+ }
+
+ private boolean transitionPowerState(OutOfBandManagement.PowerState.Event event, OutOfBandManagement outOfBandManagementHost) {
+ if (outOfBandManagementHost == null) {
+ return false;
+ }
+ OutOfBandManagement.PowerState currentPowerState = outOfBandManagementHost.getPowerState();
+ try {
+ OutOfBandManagement.PowerState newPowerState = OutOfBandManagement.PowerState.getStateMachine().getNextState(currentPowerState, event);
+ boolean result = outOfBandManagementDao.updateState(currentPowerState, event, newPowerState, outOfBandManagementHost, null);
+ if (result) {
+ final String message = String.format("Transitioned out-of-band management power state from:%s to:%s due to event:%s for the host id:%d", currentPowerState, newPowerState, event, outOfBandManagementHost.getHostId());
+ LOG.debug(message);
+ ActionEventUtils.onActionEvent(CallContext.current().getCallingUserId(), CallContext.current().getCallingAccountId(), Domain.ROOT_DOMAIN,
+ EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_POWERSTATE_TRANSITION, message);
+ }
+ return result;
+ } catch (NoTransitionException ignored) {
+ LOG.trace(String.format("Unable to transition out-of-band management power state for host id=%s for the event=%s and current power state=%s", outOfBandManagementHost.getHostId(), event, currentPowerState));
+ }
+ return false;
+ }
+
+ private boolean isOutOfBandManagementEnabledForZone(Long zoneId) {
+ if (zoneId == null) {
+ return true;
+ }
+ final DataCenterDetailVO zoneDetails = dataCenterDetailsDao.findDetail(zoneId, OOBM_ENABLED_DETAIL);
+ if (zoneDetails != null && !Strings.isNullOrEmpty(zoneDetails.getValue()) && !Boolean.valueOf(zoneDetails.getValue())) {
+ return false;
+ }
+ return true;
+ }
+
+ private boolean isOutOfBandManagementEnabledForCluster(Long clusterId) {
+ if (clusterId == null) {
+ return true;
+ }
+ final ClusterDetailsVO clusterDetails = clusterDetailsDao.findDetail(clusterId, OOBM_ENABLED_DETAIL);
+ if (clusterDetails != null && !Strings.isNullOrEmpty(clusterDetails.getValue()) && !Boolean.valueOf(clusterDetails.getValue())) {
+ return false;
+ }
+ return true;
+ }
+
+ private boolean isOutOfBandManagementEnabledForHost(Long hostId) {
+ if (hostId == null) {
+ return false;
+ }
+ final OutOfBandManagement outOfBandManagementConfig = outOfBandManagementDao.findByHost(hostId);
+ if (outOfBandManagementConfig == null || !outOfBandManagementConfig.isEnabled()) {
+ return false;
+ }
+ return true;
+ }
+
+ private void checkOutOfBandManagementEnabledByZoneClusterHost(final Host host) {
+ if (!isOutOfBandManagementEnabledForZone(host.getDataCenterId())) {
+ throw new CloudRuntimeException("Out-of-band management is disabled for the host's zone. Aborting Operation.");
+ }
+ if (!isOutOfBandManagementEnabledForCluster(host.getClusterId())) {
+ throw new CloudRuntimeException("Out-of-band management is disabled for the host's cluster. Aborting Operation.");
+ }
+ if (!isOutOfBandManagementEnabledForHost(host.getId())) {
+ throw new CloudRuntimeException("Out-of-band management is disabled or not configured for the host. Aborting Operation.");
+ }
+ }
+
+ public boolean isOutOfBandManagementEnabled(final Host host) {
+ return isOutOfBandManagementEnabledForZone(host.getDataCenterId())
+ && isOutOfBandManagementEnabledForCluster(host.getClusterId())
+ && isOutOfBandManagementEnabledForHost(host.getId());
+ }
+
+ public boolean transitionPowerStateToDisabled(List<? extends Host> hosts) {
+ boolean result = true;
+ for (Host host : hosts) {
+ result = result && transitionPowerState(OutOfBandManagement.PowerState.Event.Disabled,
+ outOfBandManagementDao.findByHost(host.getId()));
+ }
+ return result;
+ }
+
+ public void submitBackgroundPowerSyncTask(final Host host) {
+ if (host != null) {
+ // Note: This is a blocking queue based executor
+ backgroundSyncExecutor.submit(new OutOfBandManagementBackgroundTask(this, host, OutOfBandManagement.PowerOperation.STATUS));
+ }
+ }
+
+ private OutOfBandManagementResponse buildEnableDisableResponse(final boolean enabled) {
+ final OutOfBandManagementResponse response = new OutOfBandManagementResponse();
+ response.setEnabled(enabled);
+ response.setSuccess(true);
+ return response;
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_ENABLEDISABLE, eventDescription = "enabling/disabling out-of-band management on a zone")
+ public OutOfBandManagementResponse enableDisableOutOfBandManagement(final DataCenter zone, final boolean enabled) {
+ dataCenterDetailsDao.persist(zone.getId(), OOBM_ENABLED_DETAIL, String.valueOf(enabled));
+ if (!enabled) {
+ transitionPowerStateToDisabled(hostDao.findByDataCenterId(zone.getId()));
+ }
+ return buildEnableDisableResponse(enabled);
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_ENABLEDISABLE, eventDescription = "enabling/disabling out-of-band management on a cluster")
+ public OutOfBandManagementResponse enableDisableOutOfBandManagement(final Cluster cluster, final boolean enabled) {
+ clusterDetailsDao.persist(cluster.getId(), OOBM_ENABLED_DETAIL, String.valueOf(enabled));
+ if (!enabled) {
+ transitionPowerStateToDisabled(hostDao.findByClusterId(cluster.getId()));
+ }
+ return buildEnableDisableResponse(enabled);
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_ENABLEDISABLE, eventDescription = "enabling/disabling out-of-band management on a host")
+ public OutOfBandManagementResponse enableDisableOutOfBandManagement(final Host host, final boolean enabled) {
+ final OutOfBandManagement outOfBandManagementConfig = outOfBandManagementDao.findByHost(host.getId());
+ if (outOfBandManagementConfig == null) {
+ final OutOfBandManagementResponse response = new OutOfBandManagementResponse(null);
+ response.setSuccess(false);
+ response.setResultDescription("Out-of-band management is not configured for the host. Please configure the host before enabling/disabling it.");
+ return response;
+ }
+ hostAlertCache.invalidate(host.getId());
+ outOfBandManagementConfig.setEnabled(enabled);
+ boolean updateResult = outOfBandManagementDao.update(outOfBandManagementConfig.getId(), (OutOfBandManagementVO) outOfBandManagementConfig);
+ if (updateResult && !enabled) {
+ transitionPowerStateToDisabled(Collections.singletonList(host));
+ }
+ return buildEnableDisableResponse(enabled && updateResult);
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_CONFIGURE, eventDescription = "updating out-of-band management configuration")
+ public OutOfBandManagementResponse configureOutOfBandManagement(final Host host, final ImmutableMap<OutOfBandManagement.Option, String> options) {
+ OutOfBandManagement outOfBandManagementConfig = outOfBandManagementDao.findByHost(host.getId());
+ if (outOfBandManagementConfig == null) {
+ outOfBandManagementConfig = outOfBandManagementDao.persist(new OutOfBandManagementVO(host.getId()));
+ }
+ outOfBandManagementConfig = updateConfig(outOfBandManagementConfig, options);
+ if (Strings.isNullOrEmpty(outOfBandManagementConfig.getDriver()) || !outOfBandManagementDriversMap.containsKey(outOfBandManagementConfig.getDriver().toLowerCase())) {
+ throw new CloudRuntimeException("Out-of-band management driver is not available. Please provide a valid driver name.");
+ }
+
+ boolean updatedConfig = outOfBandManagementDao.update(outOfBandManagementConfig.getId(), (OutOfBandManagementVO) outOfBandManagementConfig);
+ CallContext.current().setEventDetails("host id:" + host.getId() + " configuration:" + outOfBandManagementConfig.getAddress() + ":" + outOfBandManagementConfig.getPort());
+
+ if (!updatedConfig) {
+ throw new CloudRuntimeException("Failed to update out-of-band management config for the host in the database.");
+ }
+
+ String result = "Out-of-band management successfully configured for the host";
+ LOG.debug(result);
+
+ final OutOfBandManagementResponse response = new OutOfBandManagementResponse(outOfBandManagementDao.findByHost(host.getId()));
+ response.setResultDescription(result);
+ response.setSuccess(true);
+ return response;
--- End diff --
Why not use the ``buildEnableDisableResponse`` private method here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by bhaisaab <gi...@git.apache.org>.
Github user bhaisaab commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r60716681
--- Diff: api/src/org/apache/cloudstack/api/command/admin/outofbandmanagement/ChangeOutOfBandManagementPasswordCmd.java ---
@@ -0,0 +1,116 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.api.command.admin.outofbandmanagement;
+
+import com.cloud.event.EventTypes;
+import com.cloud.exception.ConcurrentOperationException;
+import com.cloud.exception.InsufficientCapacityException;
+import com.cloud.exception.NetworkRuleConflictException;
+import com.cloud.exception.ResourceAllocationException;
+import com.cloud.exception.ResourceUnavailableException;
+import com.cloud.host.Host;
+import com.google.common.base.Strings;
+import org.apache.cloudstack.acl.RoleType;
+import org.apache.cloudstack.api.APICommand;
+import org.apache.cloudstack.api.ApiConstants;
+import org.apache.cloudstack.api.ApiErrorCode;
+import org.apache.cloudstack.api.BaseAsyncCmd;
+import org.apache.cloudstack.api.Parameter;
+import org.apache.cloudstack.api.ServerApiException;
+import org.apache.cloudstack.api.response.HostResponse;
+import org.apache.cloudstack.api.response.OutOfBandManagementResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagementService;
+
+import javax.inject.Inject;
+
+@APICommand(name = "changeOutOfBandManagementPassword", description = "Changes out-of-band management interface password on the host and updates the interface configuration in CloudStack if the operation succeeds, else reverts the old password",
+ responseObject = OutOfBandManagementResponse.class, requestHasSensitiveInfo = true, responseHasSensitiveInfo = false, authorized = {RoleType.Admin})
+public class ChangeOutOfBandManagementPasswordCmd extends BaseAsyncCmd {
+ @Inject
+ private OutOfBandManagementService outOfBandManagementService;
+
+ /////////////////////////////////////////////////////
+ //////////////// API parameters /////////////////////
+ /////////////////////////////////////////////////////
+
+ @Parameter(name = ApiConstants.HOST_ID, type = CommandType.UUID, entityType = HostResponse.class, required = true, description = "the ID of the host")
+ private Long hostId;
+
+ @Parameter(name = ApiConstants.PASSWORD, type = CommandType.STRING, description = "the new host management interface password of maximum length 16, if none is provided a random password would be used")
+ private String password;
+
+ /////////////////////////////////////////////////////
+ /////////////// API Implementation///////////////////
+ /////////////////////////////////////////////////////
+
+ @Override
+ public String getCommandName() {
+ return "changeoutofbandmanagementpasswordresponse";
+ }
+
+ private void validateParams() {
+ if (getHostId() == null || getHostId() < 1L) {
--- End diff --
I'll remove validateParams() as soon as rbac gets merged to master, I'll rebase this PR and use the new validations annotations field.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61845772
--- Diff: plugins/outofbandmanagement-drivers/ipmitool/src/org/apache/cloudstack/outofbandmanagement/driver/ipmitool/IpmitoolOutOfBandManagementDriver.java ---
@@ -0,0 +1,159 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.outofbandmanagement.driver.ipmitool;
+
+import com.cloud.utils.component.AdapterBase;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.google.common.base.Strings;
+import com.google.common.collect.ImmutableMap;
+import org.apache.cloudstack.framework.config.ConfigKey;
+import org.apache.cloudstack.framework.config.Configurable;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagement;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagementDriver;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverChangePasswordCommand;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverCommand;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverPowerCommand;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverResponse;
+import org.apache.log4j.Logger;
+
+import javax.ejb.Local;
+import java.util.Arrays;
+import java.util.List;
+
+@Local(value = {OutOfBandManagementDriver.class})
+public class IpmitoolOutOfBandManagementDriver extends AdapterBase implements OutOfBandManagementDriver, Configurable {
+ public static final Logger LOG = Logger.getLogger(IpmitoolOutOfBandManagementDriver.class);
+
+ private static volatile boolean isDriverEnabled = false;
+ private static boolean isIpmiToolBinAvailable = false;
--- End diff --
ipmitool path can be changed without restarting mgmt server, the global setting is dynamic in nature. The path is read from db for each cmd. In case ipmitool path needs tobe updated or tool need to be installed, this saves admins from restarting mgmt server. Both are static as Spring creates only one instance of this class and injects them to the core oobm service; we use them to check whether the driver is enabled and ipmitool binary is available.
On each execute, the initDriver is called to initialize the driver is ipmitool driver is not successfully initialized.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61843039
--- Diff: api/src/org/apache/cloudstack/api/command/admin/outofbandmanagement/IssueOutOfBandManagementPowerActionCmd.java ---
@@ -0,0 +1,128 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.api.command.admin.outofbandmanagement;
+
+import com.cloud.event.EventTypes;
+import com.cloud.exception.ConcurrentOperationException;
+import com.cloud.exception.InsufficientCapacityException;
+import com.cloud.exception.NetworkRuleConflictException;
+import com.cloud.exception.ResourceAllocationException;
+import com.cloud.exception.ResourceUnavailableException;
+import com.cloud.host.Host;
+import com.google.common.base.Strings;
+import org.apache.cloudstack.acl.RoleType;
+import org.apache.cloudstack.api.APICommand;
+import org.apache.cloudstack.api.ApiCommandJobType;
+import org.apache.cloudstack.api.ApiConstants;
+import org.apache.cloudstack.api.ApiErrorCode;
+import org.apache.cloudstack.api.BaseAsyncCmd;
+import org.apache.cloudstack.api.Parameter;
+import org.apache.cloudstack.api.ServerApiException;
+import org.apache.cloudstack.api.response.HostResponse;
+import org.apache.cloudstack.api.response.OutOfBandManagementResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagement.PowerOperation;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagementService;
+
+import javax.inject.Inject;
+
+@APICommand(name = "issueOutOfBandManagementPowerAction", description = "Initiates the specified power action to the host's out-of-band management interface",
--- End diff --
fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-214257913
==== Marvin Init Successful ====
=== TestName: test_oobm_background_powerstate_sync | Status : SUCCESS ===
=== TestName: test_oobm_change_password | Status : SUCCESS ===
=== TestName: test_oobm_configure_default_driver | Status : SUCCESS ===
=== TestName: test_oobm_configure_invalid_driver | Status : SUCCESS ===
=== TestName: test_oobm_disable_feature_invalid | Status : SUCCESS ===
=== TestName: test_oobm_disable_feature_valid | Status : SUCCESS ===
=== TestName: test_oobm_enable_feature_invalid | Status : SUCCESS ===
=== TestName: test_oobm_enable_feature_valid | Status : SUCCESS ===
=== TestName: test_oobm_enabledisable_across_clusterzones | Status : SUCCESS ===
=== TestName: test_oobm_issue_power_cycle | Status : SUCCESS ===
=== TestName: test_oobm_issue_power_off | Status : SUCCESS ===
=== TestName: test_oobm_issue_power_on | Status : SUCCESS ===
=== TestName: test_oobm_issue_power_reset | Status : SUCCESS ===
=== TestName: test_oobm_issue_power_soft | Status : SUCCESS ===
=== TestName: test_oobm_issue_power_status | Status : SUCCESS ===
=== TestName: test_oobm_multiple_mgmt_server_ownership | Status : SUCCESS ===
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/cloudstack/pull/1502
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61850243
--- Diff: test/integration/smoke/test_outofbandmanagement.py ---
@@ -0,0 +1,561 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import marvin
+from marvin.cloudstackTestCase import *
+from marvin.cloudstackAPI import *
+from marvin.lib.utils import *
+from marvin.lib.base import *
+from marvin.lib.common import *
+from marvin.lib.utils import (random_gen)
+from nose.plugins.attrib import attr
+
+from ipmisim.ipmisim import IpmiServerContext, IpmiServer, ThreadedIpmiServer
+
+import socket
+import sys
+import thread
+import time
+
+
+class TestOutOfBandManagement(cloudstackTestCase):
+ """ Test cases for out of band management
+ """
+
+ def setUp(self):
+ self.apiclient = self.testClient.getApiClient()
+ self.hypervisor = self.testClient.getHypervisorInfo()
+ self.dbclient = self.testClient.getDbConnection()
+ self.services = self.testClient.getParsedTestDataConfig()
+ self.mgtSvrDetails = self.config.__dict__["mgtSvr"][0].__dict__
+
+ self.zone = get_zone(self.apiclient, self.testClient.getZoneForTests())
+ self.host = None
+ self.server = None
+
+ # use random port for ipmisim
+ s = socket.socket()
+ s.bind(('', 0))
+ self.serverPort = s.getsockname()[1]
+ s.close()
+
+ self.cleanup = []
+
+
+ def tearDown(self):
+ try:
+ self.dbclient.execute("delete from oobm where port=%d" % self.getIpmiServerPort())
+ self.dbclient.execute("delete from mshost_peer where peer_runid=%s" % self.getFakeMsRunId())
+ self.dbclient.execute("delete from mshost where runid=%s" % self.getFakeMsRunId())
+ self.dbclient.execute("delete from cluster_details where name='outOfBandManagementEnabled'")
+ self.dbclient.execute("delete from data_center_details where name='outOfBandManagementEnabled'")
+ cleanup_resources(self.apiclient, self.cleanup)
+ if self.server:
+ self.server.shutdown()
+ self.server.server_close()
+ except Exception as e:
+ raise Exception("Warning: Exception during cleanup : %s" % e)
+
+
+ def getFakeMsId(self):
+ return 1234567890
+
+
+ def getFakeMsRunId(self):
+ return 123456
--- End diff --
Fixed, uses random.randint now
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61846016
--- Diff: plugins/outofbandmanagement-drivers/ipmitool/test/org/apache/cloudstack/outofbandmanagement/driver/ipmitool/IpmitoolWrapperTest.java ---
@@ -0,0 +1,107 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+
+package org.apache.cloudstack.outofbandmanagement.driver.ipmitool;
+
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.google.common.collect.ImmutableMap;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagement;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverResponse;
+import org.junit.Assert;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.mockito.runners.MockitoJUnitRunner;
+
+import java.util.Arrays;
+import java.util.List;
+
+@RunWith(MockitoJUnitRunner.class)
+public class IpmitoolWrapperTest {
+ @Test
+ public void testParsePowerCommandValid() {
+ Assert.assertEquals(IpmitoolWrapper.parsePowerCommand(OutOfBandManagement.PowerOperation.ON), "on");
+ Assert.assertEquals(IpmitoolWrapper.parsePowerCommand(OutOfBandManagement.PowerOperation.OFF), "off");
+ Assert.assertEquals(IpmitoolWrapper.parsePowerCommand(OutOfBandManagement.PowerOperation.CYCLE), "cycle");
+ Assert.assertEquals(IpmitoolWrapper.parsePowerCommand(OutOfBandManagement.PowerOperation.RESET), "reset");
+ Assert.assertEquals(IpmitoolWrapper.parsePowerCommand(OutOfBandManagement.PowerOperation.SOFT), "soft");
+ Assert.assertEquals(IpmitoolWrapper.parsePowerCommand(OutOfBandManagement.PowerOperation.STATUS), "status");
+ }
+
+ @Test
+ public void testParsePowerCommandInvalid() {
+ try {
+ IpmitoolWrapper.parsePowerCommand(null);
+ Assert.fail("IpmitoolWrapper.parsePowerCommand failed to throw exception on invalid power state");
+ } catch (IllegalStateException e) {
+ }
+ }
+
+ @Test
+ public void testParsePowerState() {
+ Assert.assertEquals(IpmitoolWrapper.parsePowerState(null), OutOfBandManagement.PowerState.Unknown);
+ Assert.assertEquals(IpmitoolWrapper.parsePowerState(""), OutOfBandManagement.PowerState.Unknown);
+ Assert.assertEquals(IpmitoolWrapper.parsePowerState(" "), OutOfBandManagement.PowerState.Unknown);
+ Assert.assertEquals(IpmitoolWrapper.parsePowerState("invalid data"), OutOfBandManagement.PowerState.Unknown);
+ Assert.assertEquals(IpmitoolWrapper.parsePowerState("Chassis Power is on"), OutOfBandManagement.PowerState.On);
+ Assert.assertEquals(IpmitoolWrapper.parsePowerState("Chassis Power is off"), OutOfBandManagement.PowerState.Off);
+ }
+
+ @Test
+ public void testGetIpmiToolCommandArgs() {
+ List<String> args = IpmitoolWrapper.getIpmiToolCommandArgs("binpath", "intf", "1", null);
+ assert args != null;
+ Assert.assertEquals(args.size(), 6);
+ Assert.assertArrayEquals(args.toArray(), new String[]{"binpath", "-I", "intf", "-R", "1", "-v"});
+
+ ImmutableMap.Builder<OutOfBandManagement.Option, String> argMap = new ImmutableMap.Builder<>();
+ argMap.put(OutOfBandManagement.Option.DRIVER, "ipmitool");
+ argMap.put(OutOfBandManagement.Option.ADDRESS, "127.0.0.1");
+ List<String> argsWithOpts = IpmitoolWrapper.getIpmiToolCommandArgs("binpath", "intf", "1", argMap.build(), "user", "list");
+ assert argsWithOpts != null;
+ Assert.assertEquals(argsWithOpts.size(), 10);
+ Assert.assertArrayEquals(argsWithOpts.toArray(), new String[]{"binpath", "-I", "intf", "-R", "1", "-v", "-H", "127.0.0.1", "user", "list"});
+ }
+
+ @Test
+ public void testFindIpmiUser() {
+ // Invalid data
+ try {
+ IpmitoolWrapper.findIpmiUser("some invalid string\n", "admin");
+ Assert.fail("IpmitoolWrapper.findIpmiUser failed to throw exception on invalid data");
+ } catch (CloudRuntimeException ignore) {
+ }
--- End diff --
Fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61843227
--- Diff: api/src/org/apache/cloudstack/api/command/admin/outofbandmanagement/IssueOutOfBandManagementPowerActionCmd.java ---
@@ -0,0 +1,128 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.api.command.admin.outofbandmanagement;
+
+import com.cloud.event.EventTypes;
+import com.cloud.exception.ConcurrentOperationException;
+import com.cloud.exception.InsufficientCapacityException;
+import com.cloud.exception.NetworkRuleConflictException;
+import com.cloud.exception.ResourceAllocationException;
+import com.cloud.exception.ResourceUnavailableException;
+import com.cloud.host.Host;
+import com.google.common.base.Strings;
+import org.apache.cloudstack.acl.RoleType;
+import org.apache.cloudstack.api.APICommand;
+import org.apache.cloudstack.api.ApiCommandJobType;
+import org.apache.cloudstack.api.ApiConstants;
+import org.apache.cloudstack.api.ApiErrorCode;
+import org.apache.cloudstack.api.BaseAsyncCmd;
+import org.apache.cloudstack.api.Parameter;
+import org.apache.cloudstack.api.ServerApiException;
+import org.apache.cloudstack.api.response.HostResponse;
+import org.apache.cloudstack.api.response.OutOfBandManagementResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagement.PowerOperation;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagementService;
+
+import javax.inject.Inject;
+
+@APICommand(name = "issueOutOfBandManagementPowerAction", description = "Initiates the specified power action to the host's out-of-band management interface",
+ responseObject = OutOfBandManagementResponse.class, requestHasSensitiveInfo = false, responseHasSensitiveInfo = false, authorized = {RoleType.Admin})
+public class IssueOutOfBandManagementPowerActionCmd extends BaseAsyncCmd {
+ @Inject
+ private OutOfBandManagementService outOfBandManagementService;
+
+ /////////////////////////////////////////////////////
+ //////////////// API parameters /////////////////////
+ /////////////////////////////////////////////////////
+
+ @Parameter(name = ApiConstants.HOST_ID, type = CommandType.UUID, entityType = HostResponse.class, required = true, description = "the ID of the host")
+ private Long hostId;
+
+ @Parameter(name = ApiConstants.TIMEOUT, type = CommandType.LONG, description = "optional operation timeout in seconds that overrides the global or cluster-level out-of-band management timeout setting")
+ private Long actionTimeout;
+
+ @Parameter(name = ApiConstants.ACTION, type = CommandType.STRING, required = true, description = "out-of-band management power actions, valid actions are: ON, OFF, CYCLE, RESET, SOFT, STATUS")
+ private String outOfBandManagementPowerOperation;
+
+ /////////////////////////////////////////////////////
+ /////////////// API Implementation///////////////////
+ /////////////////////////////////////////////////////
+
+ @Override
+ public String getCommandName() {
+ return "issueoutofbandmanagementpoweractionresponse";
+ }
+
+ private void validateParams() {
+ if (getHostId() == null || getHostId() < 1L) {
+ throw new ServerApiException(ApiErrorCode.PARAM_ERROR, "Invalid host ID: " + getHostId());
+ }
+ if (Strings.isNullOrEmpty(getOutOfBandManagementPowerOperation())) {
+ throw new ServerApiException(ApiErrorCode.PARAM_ERROR, "Invalid out-of-band management power action: " + getOutOfBandManagementPowerOperation());
+ }
--- End diff --
Fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r62612221
--- Diff: utils/src/main/java/org/apache/cloudstack/utils/process/ProcessRunner.java ---
@@ -0,0 +1,112 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+
+package org.apache.cloudstack.utils.process;
+
+import com.cloud.utils.concurrency.NamedThreadFactory;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Strings;
+import org.apache.log4j.Logger;
+import org.joda.time.Duration;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.List;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+public class ProcessRunner {
+ public static final Logger LOG = Logger.getLogger(ProcessRunner.class);
+
+ private static final ExecutorService processExecutor = Executors.newCachedThreadPool(new NamedThreadFactory("ProcessRunner"));
--- End diff --
This is a general utility, so it won't be possible to configure the executor service pool size here without passing a global setting variable on initialization. The cached pool ensures to reuse threads and still a better way than spawning separate threads on each run. Wrt oobm, for each issue command only one thread is spawned that execs an ipmitool process, a maximum of 250 (or size specified in db.properties which forces the size of max. no of concurrent async jobs) ipmitool processes can be spawned at any given time.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r62656777
--- Diff: utils/src/main/java/org/apache/cloudstack/utils/process/ProcessRunner.java ---
@@ -0,0 +1,109 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+
+package org.apache.cloudstack.utils.process;
+
+import com.cloud.utils.concurrency.NamedThreadFactory;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Strings;
+import org.apache.log4j.Logger;
+import org.joda.time.Duration;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.List;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+public class ProcessRunner {
+ public static final Logger LOG = Logger.getLogger(ProcessRunner.class);
+
+ private static final ExecutorService processExecutor = Executors.newCachedThreadPool(new NamedThreadFactory("ProcessRunner"));
+
+ private static String readStream(final InputStream inputStream) throws IOException {
+ final StringBuilder sb = new StringBuilder();
+ final BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(inputStream));
+ String line;
+ while ((line = bufferedReader.readLine()) != null) {
+ sb.append(line);
+ sb.append("\n");
+ }
+ return sb.toString();
+ }
+
+ public static ProcessResult executeCommands(final List<String> commands, final Duration timeOut) {
+ Preconditions.checkArgument(commands != null && timeOut != null);
+
+ int retVal = -2;
+ String stdOutput = null;
+ String stdError = null;
+
+ try {
+ final Process process = new ProcessBuilder().command(commands).start();
+ if (timeOut.getStandardSeconds() > 0) {
+ final Future<Integer> processFuture = processExecutor.submit(new Callable<Integer>() {
+ @Override
+ public Integer call() throws Exception {
+ return process.waitFor();
+ }
+ });
+ try {
+ retVal = processFuture.get(timeOut.getStandardSeconds(), TimeUnit.SECONDS);
+ } catch (ExecutionException e) {
+ retVal = -1;
+ stdError = e.getMessage();
+ if (LOG.isTraceEnabled()) {
+ LOG.trace("Failed to complete the requested command due to execution error: " + e.getMessage());
+ }
+ } catch (TimeoutException e) {
+ retVal = -1;
+ stdError = "Operation timed out, aborted";
+ if (LOG.isTraceEnabled()) {
+ LOG.trace("Failed to complete the requested command within timeout: " + e.getMessage());
+ }
+ } finally {
+ if (Strings.isNullOrEmpty(stdError)) {
+ stdOutput = readStream(process.getInputStream());
--- End diff --
Of course I've not, I'll play with it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61845392
--- Diff: engine/schema/src/org/apache/cloudstack/outofbandmanagement/dao/OutOfBandManagementDaoImpl.java ---
@@ -0,0 +1,163 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.outofbandmanagement.dao;
+
+import com.cloud.utils.DateUtil;
+import com.cloud.utils.db.Attribute;
+import com.cloud.utils.db.DB;
+import com.cloud.utils.db.Filter;
+import com.cloud.utils.db.GenericDaoBase;
+import com.cloud.utils.db.SearchBuilder;
+import com.cloud.utils.db.SearchCriteria;
+import com.cloud.utils.db.TransactionLegacy;
+import com.cloud.utils.db.UpdateBuilder;
+import com.cloud.utils.exception.CloudRuntimeException;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagement;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagementVO;
+import org.apache.log4j.Logger;
+import org.springframework.stereotype.Component;
+
+import javax.ejb.Local;
+import java.sql.PreparedStatement;
+import java.sql.SQLException;
+import java.util.List;
+
+@DB
+@Component
+@Local(value = {OutOfBandManagementDao.class})
+public class OutOfBandManagementDaoImpl extends GenericDaoBase<OutOfBandManagementVO, Long> implements OutOfBandManagementDao {
+ private static final Logger LOG = Logger.getLogger(OutOfBandManagementDaoImpl.class);
+
+ private SearchBuilder<OutOfBandManagementVO> HostSearch;
+ private SearchBuilder<OutOfBandManagementVO> ManagementServerSearch;
+ private SearchBuilder<OutOfBandManagementVO> OutOfBandManagementOwnerSearch;
+ private SearchBuilder<OutOfBandManagementVO> StateUpdateSearch;
+
+ private Attribute PowerStateAttr;
+ private Attribute MsIdAttr;
+ private Attribute UpdateTimeAttr;
+
+ public OutOfBandManagementDaoImpl() {
+ super();
+
+ HostSearch = createSearchBuilder();
+ HostSearch.and("hostId", HostSearch.entity().getHostId(), SearchCriteria.Op.EQ);
+ HostSearch.done();
+
+ ManagementServerSearch = createSearchBuilder();
+ ManagementServerSearch.and("server", ManagementServerSearch.entity().getManagementServerId(), SearchCriteria.Op.EQ);
+ ManagementServerSearch.done();
+
+ OutOfBandManagementOwnerSearch = createSearchBuilder();
+ OutOfBandManagementOwnerSearch.and("server", OutOfBandManagementOwnerSearch.entity().getManagementServerId(), SearchCriteria.Op.EQ);
+ OutOfBandManagementOwnerSearch.or("serverNull", OutOfBandManagementOwnerSearch.entity().getManagementServerId(), SearchCriteria.Op.NULL);
+ OutOfBandManagementOwnerSearch.done();
+
+ StateUpdateSearch = createSearchBuilder();
+ StateUpdateSearch.and("status", StateUpdateSearch.entity().getPowerState(), SearchCriteria.Op.EQ);
+ StateUpdateSearch.and("id", StateUpdateSearch.entity().getId(), SearchCriteria.Op.EQ);
+ StateUpdateSearch.and("update", StateUpdateSearch.entity().getUpdateCount(), SearchCriteria.Op.EQ);
+ StateUpdateSearch.done();
+
+ PowerStateAttr = _allAttributes.get("powerState");
+ MsIdAttr = _allAttributes.get("managementServerId");
+ UpdateTimeAttr = _allAttributes.get("updateTime");
+ assert (PowerStateAttr != null && MsIdAttr != null && UpdateTimeAttr != null) : "Couldn't find one of these attributes";
+ }
+
+ @Override
+ public OutOfBandManagement findByHost(long hostId) {
+ SearchCriteria<OutOfBandManagementVO> sc = HostSearch.create("hostId", hostId);
+ return findOneBy(sc);
+ }
+
+ @Override
+ public List<OutOfBandManagementVO> findAllByManagementServer(long serverId) {
+ SearchCriteria<OutOfBandManagementVO> sc = OutOfBandManagementOwnerSearch.create();
+ sc.setParameters("server", serverId);
+ return listBy(sc, new Filter(OutOfBandManagementVO.class, "updateTime", true, null, null));
+ }
+
+ private void executeExpireOwnershipSql(final String sql, long resource) {
+ TransactionLegacy txn = TransactionLegacy.currentTxn();
+ try {
+ txn.start();
+ PreparedStatement pstmt = txn.prepareAutoCloseStatement(sql);
+ pstmt.setLong(1, resource);
+ pstmt.executeUpdate();
+ txn.commit();
+ } catch (SQLException e) {
+ txn.rollback();
+ throw new CloudRuntimeException("Unable to reset out-of-band management ownership based on resource:" + resource);
+ }
+ }
+
+ @Override
+ public void expireOutOfBandManagementOwnershipByHours(long hours) {
+ final String resetOwnerSql = "UPDATE oobm set mgmt_server_id=NULL where update_time<= (NOW() - INTERVAL ? HOUR)";
+ executeExpireOwnershipSql(resetOwnerSql, hours);
+ }
+
+ @Override
+ public void expireOutOfBandManagementOwnershipByServer(long serverId) {
+ final String resetOwnerSql = "UPDATE oobm set mgmt_server_id=NULL, power_state=NULL where mgmt_server_id=?";
+ executeExpireOwnershipSql(resetOwnerSql, serverId);
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Expired out-of-band management ownership for hosts owned by management server id:" + serverId);
+ }
+ }
+
+ @Override
+ public boolean updateState(OutOfBandManagement.PowerState oldStatus, OutOfBandManagement.PowerState.Event event, OutOfBandManagement.PowerState newStatus, OutOfBandManagement vo, Object data) {
+ // lock target row from beginning to avoid lock-promotion caused deadlock
+ OutOfBandManagementVO oobmHost = lockRow(vo.getId(), true);
+ if (oobmHost == null) {
+ if (LOG.isTraceEnabled()) {
+ LOG.trace("Failed to lock row to update state for out-of-band management host id: " + vo.getHostId());
+ }
+ return false;
+ }
+
+ Long newManagementServerId = event.getServerId();
+ // Avoid updates when old ownership and state are same as new
+ if (oldStatus == newStatus && (oobmHost.getManagementServerId() != null && oobmHost.getManagementServerId().equals(newManagementServerId))) {
+ return false;
+ }
+
+ if (event == OutOfBandManagement.PowerState.Event.Disabled) {
+ newManagementServerId = null;
+ }
+
+ SearchCriteria<OutOfBandManagementVO> sc = StateUpdateSearch.create();
+ sc.setParameters("status", oldStatus);
+ sc.setParameters("id", oobmHost.getId());
+ sc.setParameters("update", oobmHost.getUpdateCount());
+
+ oobmHost.incrUpdateCount();
--- End diff --
It's not a pessimistic lock, lockRows uses a `for update` http://dev.mysql.com/doc/refman/5.7/en/innodb-locking-reads.html
While incrUpdateCount uses a state based update counter CRDT https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type#State-based_increment-only_counter
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61817458
--- Diff: test/integration/smoke/test_outofbandmanagement.py ---
@@ -0,0 +1,561 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import marvin
+from marvin.cloudstackTestCase import *
+from marvin.cloudstackAPI import *
+from marvin.lib.utils import *
+from marvin.lib.base import *
+from marvin.lib.common import *
+from marvin.lib.utils import (random_gen)
+from nose.plugins.attrib import attr
+
+from ipmisim.ipmisim import IpmiServerContext, IpmiServer, ThreadedIpmiServer
+
+import socket
+import sys
+import thread
+import time
+
+
+class TestOutOfBandManagement(cloudstackTestCase):
+ """ Test cases for out of band management
+ """
+
+ def setUp(self):
+ self.apiclient = self.testClient.getApiClient()
+ self.hypervisor = self.testClient.getHypervisorInfo()
+ self.dbclient = self.testClient.getDbConnection()
+ self.services = self.testClient.getParsedTestDataConfig()
+ self.mgtSvrDetails = self.config.__dict__["mgtSvr"][0].__dict__
+
+ self.zone = get_zone(self.apiclient, self.testClient.getZoneForTests())
+ self.host = None
+ self.server = None
+
+ # use random port for ipmisim
+ s = socket.socket()
+ s.bind(('', 0))
+ self.serverPort = s.getsockname()[1]
+ s.close()
+
+ self.cleanup = []
+
+
+ def tearDown(self):
+ try:
+ self.dbclient.execute("delete from oobm where port=%d" % self.getIpmiServerPort())
+ self.dbclient.execute("delete from mshost_peer where peer_runid=%s" % self.getFakeMsRunId())
+ self.dbclient.execute("delete from mshost where runid=%s" % self.getFakeMsRunId())
+ self.dbclient.execute("delete from cluster_details where name='outOfBandManagementEnabled'")
+ self.dbclient.execute("delete from data_center_details where name='outOfBandManagementEnabled'")
+ cleanup_resources(self.apiclient, self.cleanup)
+ if self.server:
+ self.server.shutdown()
+ self.server.server_close()
+ except Exception as e:
+ raise Exception("Warning: Exception during cleanup : %s" % e)
+
+
+ def getFakeMsId(self):
+ return 1234567890
+
+
+ def getFakeMsRunId(self):
+ return 123456
+
+
+ def getHost(self, hostId=None):
+ if self.host and hostId is None:
+ return self.host
+
+ response = list_hosts(
+ self.apiclient,
+ zoneid=self.zone.id,
+ type='Routing',
+ id=hostId
+ )
+ if len(response) > 0:
+ self.host = response[0]
+ return self.host
+ raise self.skipTest("No hosts found, skipping out-of-band management test")
+
+
+ def getIpmiServerIp(self):
+ s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
+ s.connect((self.mgtSvrDetails["mgtSvrIp"], self.mgtSvrDetails["port"]))
+ return s.getsockname()[0]
+
+
+ def getIpmiServerPort(self):
+ return self.serverPort
+
+
+ def getOobmConfigCmd(self):
+ cmd = configureOutOfBandManagement.configureOutOfBandManagementCmd()
+ cmd.driver = 'ipmitool' # The default available driver
+ cmd.address = self.getIpmiServerIp()
+ cmd.port = self.getIpmiServerPort()
+ cmd.username = 'admin'
+ cmd.password = 'password'
+ cmd.hostid = self.getHost().id
+ return cmd
+
+
+ def getOobmEnableCmd(self):
+ cmd = enableOutOfBandManagementForHost.enableOutOfBandManagementForHostCmd()
+ cmd.hostid = self.getHost().id
+ return cmd
+
+
+ def getOobmDisableCmd(self):
+ cmd = disableOutOfBandManagementForHost.disableOutOfBandManagementForHostCmd()
+ cmd.hostid = self.getHost().id
+ return cmd
+
+
+ def getOobmIssueActionCmd(self):
+ cmd = issueOutOfBandManagementPowerAction.issueOutOfBandManagementPowerActionCmd()
+ cmd.hostid = self.getHost().id
+ cmd.action = 'STATUS'
+ return cmd
+
+
+ def issuePowerActionCmd(self, action, timeout=None):
+ cmd = self.getOobmIssueActionCmd()
+ cmd.action = action
+ if timeout:
+ cmd.timeout = timeout
+ return self.apiclient.issueOutOfBandManagementPowerAction(cmd)
+
+
+ def configureAndEnableOobm(self):
+ self.apiclient.configureOutOfBandManagement(self.getOobmConfigCmd())
+ response = self.apiclient.enableOutOfBandManagementForHost(self.getOobmEnableCmd())
+ self.assertEqual(response.enabled, True)
+
+
+ def startIpmiServer(self):
+ def startIpmiServer(tname, server):
+ self.debug("Starting ipmisim server")
+ try:
+ server.serve_forever()
+ except Exception: pass
+ IpmiServerContext('reset')
+ ThreadedIpmiServer.allow_reuse_address = False
+ server = ThreadedIpmiServer(('0.0.0.0', self.getIpmiServerPort()), IpmiServer)
+ thread.start_new_thread(startIpmiServer, ("ipmi-server", server,))
+ self.server = server
+
+
+ def checkSyncToState(self, state, interval):
+ self.debug("Waiting for background thread to update powerstate to " + state)
+ time.sleep(1 + int(interval)*2/1000) # interval is in ms
+ response = self.getHost(hostId=self.getHost().id).outofbandmanagement
+ self.assertEqual(response.powerstate, state)
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_configure_invalid_driver(self):
+ """
+ Tests out-of-band management configuration with invalid driver
+ """
+ cmd = self.getOobmConfigCmd()
+ cmd.driver = 'randomDriverThatDoesNotExist'
+ try:
+ response = self.apiclient.configureOutOfBandManagement(cmd)
+ self.fail("Expected an exception to be thrown, failing")
+ except Exception: pass
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_configure_default_driver(self):
+ """
+ Tests out-of-band management configuration with valid data
+ """
+ cmd = self.getOobmConfigCmd()
+ response = self.apiclient.configureOutOfBandManagement(cmd)
+ self.assertEqual(response.hostid, cmd.hostid)
+ self.assertEqual(response.driver, cmd.driver)
+ self.assertEqual(response.address, cmd.address)
+ self.assertEqual(response.port, str(cmd.port))
+ self.assertEqual(response.username, cmd.username)
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_enable_feature_invalid(self):
+ """
+ Tests out-of-band management host enable feature with
+ invalid options
+ """
+ cmd = self.getOobmEnableCmd()
+ cmd.hostid = -1
+ try:
+ response = self.apiclient.enableOutOfBandManagementForHost(cmd)
+ self.fail("Expected an exception to be thrown, failing")
+ except Exception: pass
+
+ try:
+ cmd = enableOutOfBandManagementForCluster.enableOutOfBandManagementForClusterCmd()
+ response = self.apiclient.enableOutOfBandManagementForCluster(cmd)
+ self.fail("Expected an exception to be thrown, failing")
+ except Exception: pass
+
+ try:
+ cmd = enableOutOfBandManagementForZone.enableOutOfBandManagementForZoneCmd()
+ response = self.apiclient.enableOutOfBandManagementForZone(cmd)
+ self.fail("Expected an exception to be thrown, failing")
+ except Exception: pass
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_disable_feature_invalid(self):
+ """
+ Tests out-of-band management host disable feature with
+ invalid options
+ """
+ cmd = self.getOobmDisableCmd()
+ cmd.hostid = -1
+ try:
+ response = self.apiclient.disableOutOfBandManagementForHost(cmd)
+ self.fail("Expected an exception to be thrown, failing")
+ except Exception: pass
+
+ try:
+ cmd = disableOutOfBandManagementForCluster.disableOutOfBandManagementForClusterCmd()
+ response = self.apiclient.disableOutOfBandManagementForCluster(cmd)
+ self.fail("Expected an exception to be thrown, failing")
+ except Exception: pass
+
+ try:
+ cmd = disableOutOfBandManagementForZone.disableOutOfBandManagementForZoneCmd()
+ response = self.apiclient.disableOutOfBandManagementForZone(cmd)
+ self.fail("Expected an exception to be thrown, failing")
+ except Exception: pass
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_enable_feature_valid(self):
+ """
+ Tests out-of-band management host enable feature with
+ valid options
+ """
+ self.apiclient.configureOutOfBandManagement(self.getOobmConfigCmd())
+ cmd = self.getOobmEnableCmd()
+ response = self.apiclient.enableOutOfBandManagementForHost(cmd)
+ self.assertEqual(response.hostid, cmd.hostid)
+ self.assertEqual(response.enabled, True)
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_disable_feature_valid(self):
+ """
+ Tests out-of-band management host disable feature with
+ valid options
+ """
+
+ self.apiclient.configureOutOfBandManagement(self.getOobmConfigCmd())
+ cmd = self.getOobmDisableCmd()
+ response = self.apiclient.disableOutOfBandManagementForHost(cmd)
+ self.assertEqual(response.hostid, cmd.hostid)
+ self.assertEqual(response.enabled, False)
+
+ response = self.getHost(hostId=cmd.hostid).outofbandmanagement
+ self.assertEqual(response.powerstate, 'Disabled')
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_enabledisable_across_clusterzones(self):
+ """
+ Tests out-of-band management enable/disable feature at cluster
+ and zone level sequentially Zone > Cluster > Host
+ """
+ self.configureAndEnableOobm()
+ self.startIpmiServer()
+ bmc = IpmiServerContext().bmc
+ bmc.powerstate = 'off'
+
+ host = self.getHost()
+
+ # Disable at zone level
+ cmd = disableOutOfBandManagementForZone.disableOutOfBandManagementForZoneCmd()
+ cmd.zoneid = host.zoneid
+ response = self.apiclient.disableOutOfBandManagementForZone(cmd)
+
+ # Disable at cluster level
+ cmd = disableOutOfBandManagementForCluster.disableOutOfBandManagementForClusterCmd()
+ cmd.clusterid = host.clusterid
+ response = self.apiclient.disableOutOfBandManagementForCluster(cmd)
+
+ # Disable at host level
+ cmd = disableOutOfBandManagementForHost.disableOutOfBandManagementForHostCmd()
+ cmd.hostid = host.id
+ response = self.apiclient.disableOutOfBandManagementForHost(cmd)
+
+ try:
+ self.issuePowerActionCmd('STATUS')
+ self.fail("Exception was expected, oobm is disabled at zone level")
+ except Exception: pass
+
+ # Enable at zone level
+ cmd = enableOutOfBandManagementForZone.enableOutOfBandManagementForZoneCmd()
+ cmd.zoneid = host.zoneid
+ response = self.apiclient.enableOutOfBandManagementForZone(cmd)
+
+ try:
+ self.issuePowerActionCmd('STATUS')
+ self.fail("Exception was expected, oobm is disabled at cluster level")
+ except Exception: pass
+
+ # Check background thread syncs state to Disabled
+ response = self.getHost(hostId=host.id).outofbandmanagement
+ self.assertEqual(response.powerstate, 'Disabled')
+ self.dbclient.execute("update oobm set power_state='On' where port=%d" % self.getIpmiServerPort())
+ interval = list_configurations(
+ self.apiclient,
+ name='outofbandmanagement.sync.interval'
+ )[0].value
+ self.checkSyncToState('Disabled', interval)
+
+ # Enable at cluster level
+ cmd = enableOutOfBandManagementForCluster.enableOutOfBandManagementForClusterCmd()
+ cmd.clusterid = host.clusterid
+ response = self.apiclient.enableOutOfBandManagementForCluster(cmd)
+
+ try:
+ self.issuePowerActionCmd('STATUS')
+ self.fail("Exception was expected, oobm is disabled at host level")
+ except Exception: pass
+
+ # Enable at host level
+ cmd = enableOutOfBandManagementForHost.enableOutOfBandManagementForHostCmd()
+ cmd.hostid = host.id
+ response = self.apiclient.enableOutOfBandManagementForHost(cmd)
+
+ response = self.issuePowerActionCmd('STATUS')
+ self.assertEqual(response.powerstate, 'Off')
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_issue_power_status(self):
+ """
+ Tests out-of-band management issue power action
+ """
+ self.configureAndEnableOobm()
+ self.startIpmiServer()
+ bmc = IpmiServerContext().bmc
+ bmc.powerstate = 'on'
+ response = self.issuePowerActionCmd('STATUS')
+ self.assertEqual(response.powerstate, 'On')
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_issue_power_on(self):
+ """
+ Tests out-of-band management issue power on action
+ """
+ self.configureAndEnableOobm()
+ self.startIpmiServer()
+ self.issuePowerActionCmd('ON')
+ response = self.issuePowerActionCmd('STATUS')
+ self.assertEqual(response.powerstate, 'On')
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_issue_power_off(self):
+ """
+ Tests out-of-band management issue power off action
+ """
+ self.configureAndEnableOobm()
+ self.startIpmiServer()
+ self.issuePowerActionCmd('OFF')
+ response = self.issuePowerActionCmd('STATUS')
+ self.assertEqual(response.powerstate, 'Off')
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_issue_power_cycle(self):
+ """
+ Tests out-of-band management issue power cycle action
+ """
+ self.configureAndEnableOobm()
+ self.startIpmiServer()
+ self.issuePowerActionCmd('CYCLE')
+ response = self.issuePowerActionCmd('STATUS')
+ self.assertEqual(response.powerstate, 'On')
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_issue_power_reset(self):
+ """
+ Tests out-of-band management issue power reset action
+ """
+ self.configureAndEnableOobm()
+ self.startIpmiServer()
+ self.issuePowerActionCmd('RESET')
+ response = self.issuePowerActionCmd('STATUS')
+ self.assertEqual(response.powerstate, 'On')
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_issue_power_soft(self):
+ """
+ Tests out-of-band management issue power soft action
+ """
+ self.configureAndEnableOobm()
+ self.startIpmiServer()
+ self.issuePowerActionCmd('SOFT')
+ response = self.issuePowerActionCmd('STATUS')
+ self.assertEqual(response.powerstate, 'Off')
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_background_powerstate_sync(self):
+ """
+ Tests out-of-band management background powerstate sync
+ """
+ self.debug("Testing oobm background sync")
+ interval = list_configurations(
+ self.apiclient,
+ name='outofbandmanagement.sync.interval'
+ )[0].value
+
+ self.configureAndEnableOobm()
+ self.startIpmiServer()
+ bmc = IpmiServerContext().bmc
+
+ bmc.powerstate = 'on'
+ self.checkSyncToState('On', interval)
+
+ bmc.powerstate = 'off'
+ self.checkSyncToState('Off', interval)
+
+ self.server.shutdown()
+ self.server.server_close()
+
+ # Check for unknown state (ipmi server not reachable)
+ self.checkSyncToState('Unknown', interval)
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_multiple_mgmt_server_ownership(self):
--- End diff --
Do we have a test runs multiple management servers and attempts to fail one?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61814453
--- Diff: server/test/org/apache/cloudstack/outofbandmanagement/OutOfBandManagementServiceTest.java ---
@@ -0,0 +1,119 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.outofbandmanagement;
+
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.google.common.collect.ImmutableMap;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverResponse;
+import org.junit.Assert;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.mockito.runners.MockitoJUnitRunner;
+
+@RunWith(MockitoJUnitRunner.class)
+public class OutOfBandManagementServiceTest {
+
+ OutOfBandManagementServiceImpl oobmService = new OutOfBandManagementServiceImpl();
+
+ @Test
+ public void testOutOfBandManagementDriverResponseEvent() {
+ OutOfBandManagementDriverResponse r = new OutOfBandManagementDriverResponse("some result", "some error", false);
+
+ r.setSuccess(false);
+ r.setAuthFailure(false);
+ Assert.assertEquals(r.toEvent(), OutOfBandManagement.PowerState.Event.Unknown);
+
+ r.setSuccess(false);
+ r.setAuthFailure(true);
+ Assert.assertEquals(r.toEvent(), OutOfBandManagement.PowerState.Event.AuthError);
+
+ r.setAuthFailure(false);
+ r.setSuccess(true);
+ r.setPowerState(OutOfBandManagement.PowerState.On);
+ Assert.assertEquals(r.toEvent(), OutOfBandManagement.PowerState.Event.On);
+
+ r.setPowerState(OutOfBandManagement.PowerState.Off);
+ Assert.assertEquals(r.toEvent(), OutOfBandManagement.PowerState.Event.Off);
+
+ r.setPowerState(OutOfBandManagement.PowerState.Disabled);
+ Assert.assertEquals(r.toEvent(), OutOfBandManagement.PowerState.Event.Disabled);
+ }
+
+ private ImmutableMap<OutOfBandManagement.Option, String> buildRandomOptionsMap() {
+ ImmutableMap.Builder<OutOfBandManagement.Option, String> builder = new ImmutableMap.Builder<>();
+ builder.put(OutOfBandManagement.Option.ADDRESS, "localhost");
+ builder.put(OutOfBandManagement.Option.DRIVER, "ipmitool");
+ return builder.build();
+ }
+
+ @Test
+ public void testUpdateOutOfBandManagementConfigValid() {
+ OutOfBandManagement config = new OutOfBandManagementVO(123L);
+ Assert.assertEquals(config.getPowerState(), OutOfBandManagement.PowerState.Disabled);
+ config = oobmService.updateConfig(config, buildRandomOptionsMap());
+ Assert.assertEquals(config.getAddress(), "localhost");
+ Assert.assertEquals(config.getDriver(), "ipmitool");
+ Assert.assertEquals(config.getPowerState(), OutOfBandManagement.PowerState.Disabled);
+ }
+
+ @Test
+ public void testUpdateOutOfBandManagementConfigInValid() {
+ try {
+ oobmService.updateConfig(null, buildRandomOptionsMap());
+ Assert.fail("CloudRuntimeException was expect for out-of-band management not configured for the host");
+ } catch (CloudRuntimeException ignored) {
+ }
+
+ try {
+ oobmService.updateConfig(null, null);
+ Assert.fail("CloudRuntimeException was expect for out-of-band management not configured for the host");
+ } catch (CloudRuntimeException ignored) {
+ }
+
+ OutOfBandManagement config = new OutOfBandManagementVO(123L);
+ config.setAddress(null);
+ config = oobmService.updateConfig(config, null);
+ Assert.assertEquals(config.getAddress(), null);
+ Assert.assertEquals(config.getPowerState(), OutOfBandManagement.PowerState.Disabled);
+ }
+
+ @Test
+ public void testGetOutOfBandManagementOptionsValid() {
+ OutOfBandManagement configEmpty = new OutOfBandManagementVO(123L);
+ ImmutableMap<OutOfBandManagement.Option, String> optionsEmpty = oobmService.getOptions(configEmpty);
+ Assert.assertEquals(optionsEmpty.size(), 0);
+
+ OutOfBandManagement config = new OutOfBandManagementVO(123L);
+ config.setAddress("localhost");
+ config.setDriver("ipmitool");
+ config.setPort(1234);
+ ImmutableMap<OutOfBandManagement.Option, String> options = oobmService.getOptions(config);
+ Assert.assertEquals(options.get(OutOfBandManagement.Option.ADDRESS), "localhost");
+ Assert.assertEquals(options.get(OutOfBandManagement.Option.DRIVER), "ipmitool");
+ Assert.assertEquals(options.get(OutOfBandManagement.Option.PORT), "1234");
+ }
+
+ @Test
+ public void testGetOutOfBandManagementOptionsInvalid() {
+ try {
+ oobmService.getOptions(null);
+ Assert.fail("CloudRuntimeException was expected for finding options of host with out-of-band management configuration");
+ } catch (CloudRuntimeException e) {
+ }
--- End diff --
This ``try`` block and ``fail`` construct is unnecessary. Changing ``@Test`` to ``@Test(expected = CloudRuntimeException.class)`` accomplishes the same thing.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61847192
--- Diff: server/src/org/apache/cloudstack/outofbandmanagement/OutOfBandManagementServiceImpl.java ---
@@ -0,0 +1,532 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.outofbandmanagement;
+
+import com.cloud.alert.AlertManager;
+import com.cloud.dc.ClusterDetailsDao;
+import com.cloud.dc.ClusterDetailsVO;
+import com.cloud.dc.DataCenter;
+import com.cloud.dc.DataCenterDetailVO;
+import com.cloud.dc.dao.DataCenterDetailsDao;
+import com.cloud.domain.Domain;
+import com.cloud.event.ActionEvent;
+import com.cloud.event.ActionEventUtils;
+import com.cloud.event.EventTypes;
+import com.cloud.host.Host;
+import com.cloud.host.dao.HostDao;
+import com.cloud.org.Cluster;
+import com.cloud.utils.component.Manager;
+import com.cloud.utils.component.ManagerBase;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.db.Transaction;
+import com.cloud.utils.db.TransactionCallback;
+import com.cloud.utils.db.TransactionStatus;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.fsm.NoTransitionException;
+import com.google.common.base.Strings;
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import com.google.common.collect.ImmutableMap;
+import org.apache.cloudstack.api.response.OutOfBandManagementResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.framework.config.ConfigKey;
+import org.apache.cloudstack.framework.config.Configurable;
+import org.apache.cloudstack.outofbandmanagement.dao.OutOfBandManagementDao;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverChangePasswordCommand;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverPowerCommand;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverResponse;
+import org.apache.cloudstack.utils.identity.ManagementServerNode;
+import org.apache.log4j.Logger;
+import org.springframework.stereotype.Component;
+
+import javax.ejb.Local;
+import javax.inject.Inject;
+import javax.naming.ConfigurationException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+@Component
+@Local(value = {OutOfBandManagementService.class})
+public class OutOfBandManagementServiceImpl extends ManagerBase implements OutOfBandManagementService, Manager, Configurable {
+ public static final Logger LOG = Logger.getLogger(OutOfBandManagementServiceImpl.class);
+
+ @Inject
+ private ClusterDetailsDao clusterDetailsDao;
+ @Inject
+ private DataCenterDetailsDao dataCenterDetailsDao;
+ @Inject
+ private OutOfBandManagementDao outOfBandManagementDao;
+ @Inject
+ private HostDao hostDao;
+ @Inject
+ private AlertManager alertMgr;
+
+ private String name;
+ private long serviceId;
+
+ private List<OutOfBandManagementDriver> outOfBandManagementDrivers = new ArrayList<>();
+ private Map<String, OutOfBandManagementDriver> outOfBandManagementDriversMap = new HashMap<String, OutOfBandManagementDriver>();
+
+ private static final String OOBM_ENABLED_DETAIL = "outOfBandManagementEnabled";
+ private static final int ACQUIRE_GLOBAL_LOCK_TIMEOUT_FOR_HOST = 120;
+
+ private Cache<Long, Long> hostAlertCache;
+ private static ExecutorService backgroundSyncExecutor;
+
+ private String getOutOfBandManagementHostLock(long id) {
+ return "oobm.host." + id;
+ }
+
+ private void initializeDriversMap() {
+ if (outOfBandManagementDriversMap.isEmpty() && outOfBandManagementDrivers != null && outOfBandManagementDrivers.size() > 0) {
+ for (final OutOfBandManagementDriver driver : outOfBandManagementDrivers) {
+ outOfBandManagementDriversMap.put(driver.getName().toLowerCase(), driver);
+ }
+ LOG.debug("Discovered out-of-band management drivers configured in the OutOfBandManagementService");
+ }
+ }
+
+ private OutOfBandManagementDriver getDriver(final OutOfBandManagement outOfBandManagementConfig) {
+ if (!Strings.isNullOrEmpty(outOfBandManagementConfig.getDriver())) {
+ final OutOfBandManagementDriver driver = outOfBandManagementDriversMap.get(outOfBandManagementConfig.getDriver());
+ if (driver != null) {
+ return driver;
+ }
+ }
+ throw new CloudRuntimeException("Configured out-of-band management driver is not available. Aborting any out-of-band management action.");
+ }
+
+ protected OutOfBandManagement updateConfig(final OutOfBandManagement outOfBandManagementConfig, final ImmutableMap<OutOfBandManagement.Option, String> options) {
+ if (outOfBandManagementConfig == null) {
+ throw new CloudRuntimeException("Out-of-band management is not configured for the host. Aborting.");
+ }
+ if (options == null) {
+ return outOfBandManagementConfig;
+ }
+ for (OutOfBandManagement.Option option: options.keySet()) {
+ final String value = options.get(option);
+ if (Strings.isNullOrEmpty(value)) {
+ continue;
+ }
+ switch (option) {
+ case DRIVER:
+ outOfBandManagementConfig.setDriver(value);
+ break;
+ case ADDRESS:
+ outOfBandManagementConfig.setAddress(value);
+ break;
+ case PORT:
+ outOfBandManagementConfig.setPort(Integer.parseInt(value));
+ break;
+ case USERNAME:
+ outOfBandManagementConfig.setUsername(value);
+ break;
+ case PASSWORD:
+ outOfBandManagementConfig.setPassword(value);
+ break;
+ }
+ }
+ return outOfBandManagementConfig;
+ }
+
+ protected ImmutableMap<OutOfBandManagement.Option, String> getOptions(final OutOfBandManagement outOfBandManagementConfig) {
+ final ImmutableMap.Builder<OutOfBandManagement.Option, String> optionsBuilder = ImmutableMap.builder();
+ if (outOfBandManagementConfig == null) {
+ throw new CloudRuntimeException("Out-of-band management is not configured for the host. Aborting.");
+ }
+ for (OutOfBandManagement.Option option: OutOfBandManagement.Option.values()) {
+ String value = null;
+ switch (option) {
+ case DRIVER:
+ value = outOfBandManagementConfig.getDriver();
+ break;
+ case ADDRESS:
+ value = outOfBandManagementConfig.getAddress();
+ break;
+ case PORT:
+ if (outOfBandManagementConfig.getPort() != null) {
+ value = String.valueOf(outOfBandManagementConfig.getPort());
+ }
+ break;
+ case USERNAME:
+ value = outOfBandManagementConfig.getUsername();
+ break;
+ case PASSWORD:
+ value = outOfBandManagementConfig.getPassword();
+ break;
+ }
+ if (value != null) {
+ optionsBuilder.put(option, value);
+ }
+ }
+ return optionsBuilder.build();
+ }
+
+ private void sendAuthError(final Host host, final String message) {
+ try {
+ hostAlertCache.asMap().putIfAbsent(host.getId(), 0L);
+ Long sentCount = hostAlertCache.asMap().get(host.getId());
+ if (sentCount != null && sentCount <= 0) {
+ boolean concurrentUpdateResult = hostAlertCache.asMap().replace(host.getId(), sentCount, sentCount+1L);
+ if (concurrentUpdateResult) {
+ final String subject = String.format("Out-of-band management auth-error detected for host:%d in cluster:%d, zone:%d", host.getId(), host.getClusterId(), host.getDataCenterId());
+ LOG.error(subject + ": " + message);
+ alertMgr.sendAlert(AlertManager.AlertType.ALERT_TYPE_OOBM_AUTH_ERROR, host.getDataCenterId(), host.getPodId(), subject, message);
+ }
+ }
+ } catch (Exception ignored) {
+ }
+ }
+
+ private boolean transitionPowerState(OutOfBandManagement.PowerState.Event event, OutOfBandManagement outOfBandManagementHost) {
+ if (outOfBandManagementHost == null) {
+ return false;
+ }
+ OutOfBandManagement.PowerState currentPowerState = outOfBandManagementHost.getPowerState();
+ try {
+ OutOfBandManagement.PowerState newPowerState = OutOfBandManagement.PowerState.getStateMachine().getNextState(currentPowerState, event);
+ boolean result = outOfBandManagementDao.updateState(currentPowerState, event, newPowerState, outOfBandManagementHost, null);
+ if (result) {
+ final String message = String.format("Transitioned out-of-band management power state from:%s to:%s due to event:%s for the host id:%d", currentPowerState, newPowerState, event, outOfBandManagementHost.getHostId());
+ LOG.debug(message);
+ ActionEventUtils.onActionEvent(CallContext.current().getCallingUserId(), CallContext.current().getCallingAccountId(), Domain.ROOT_DOMAIN,
+ EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_POWERSTATE_TRANSITION, message);
+ }
+ return result;
+ } catch (NoTransitionException ignored) {
+ LOG.trace(String.format("Unable to transition out-of-band management power state for host id=%s for the event=%s and current power state=%s", outOfBandManagementHost.getHostId(), event, currentPowerState));
+ }
+ return false;
+ }
+
+ private boolean isOutOfBandManagementEnabledForZone(Long zoneId) {
+ if (zoneId == null) {
+ return true;
+ }
+ final DataCenterDetailVO zoneDetails = dataCenterDetailsDao.findDetail(zoneId, OOBM_ENABLED_DETAIL);
+ if (zoneDetails != null && !Strings.isNullOrEmpty(zoneDetails.getValue()) && !Boolean.valueOf(zoneDetails.getValue())) {
+ return false;
+ }
+ return true;
+ }
+
+ private boolean isOutOfBandManagementEnabledForCluster(Long clusterId) {
+ if (clusterId == null) {
+ return true;
+ }
+ final ClusterDetailsVO clusterDetails = clusterDetailsDao.findDetail(clusterId, OOBM_ENABLED_DETAIL);
+ if (clusterDetails != null && !Strings.isNullOrEmpty(clusterDetails.getValue()) && !Boolean.valueOf(clusterDetails.getValue())) {
+ return false;
+ }
+ return true;
+ }
+
+ private boolean isOutOfBandManagementEnabledForHost(Long hostId) {
+ if (hostId == null) {
+ return false;
+ }
+ final OutOfBandManagement outOfBandManagementConfig = outOfBandManagementDao.findByHost(hostId);
+ if (outOfBandManagementConfig == null || !outOfBandManagementConfig.isEnabled()) {
+ return false;
+ }
+ return true;
+ }
+
+ private void checkOutOfBandManagementEnabledByZoneClusterHost(final Host host) {
+ if (!isOutOfBandManagementEnabledForZone(host.getDataCenterId())) {
+ throw new CloudRuntimeException("Out-of-band management is disabled for the host's zone. Aborting Operation.");
+ }
+ if (!isOutOfBandManagementEnabledForCluster(host.getClusterId())) {
+ throw new CloudRuntimeException("Out-of-band management is disabled for the host's cluster. Aborting Operation.");
+ }
+ if (!isOutOfBandManagementEnabledForHost(host.getId())) {
+ throw new CloudRuntimeException("Out-of-band management is disabled or not configured for the host. Aborting Operation.");
+ }
+ }
+
+ public boolean isOutOfBandManagementEnabled(final Host host) {
+ return isOutOfBandManagementEnabledForZone(host.getDataCenterId())
+ && isOutOfBandManagementEnabledForCluster(host.getClusterId())
+ && isOutOfBandManagementEnabledForHost(host.getId());
+ }
+
+ public boolean transitionPowerStateToDisabled(List<? extends Host> hosts) {
+ boolean result = true;
+ for (Host host : hosts) {
+ result = result && transitionPowerState(OutOfBandManagement.PowerState.Event.Disabled,
+ outOfBandManagementDao.findByHost(host.getId()));
+ }
+ return result;
+ }
+
+ public void submitBackgroundPowerSyncTask(final Host host) {
+ if (host != null) {
+ // Note: This is a blocking queue based executor
+ backgroundSyncExecutor.submit(new OutOfBandManagementBackgroundTask(this, host, OutOfBandManagement.PowerOperation.STATUS));
+ }
+ }
+
+ private OutOfBandManagementResponse buildEnableDisableResponse(final boolean enabled) {
+ final OutOfBandManagementResponse response = new OutOfBandManagementResponse();
+ response.setEnabled(enabled);
+ response.setSuccess(true);
+ return response;
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_ENABLEDISABLE, eventDescription = "enabling/disabling out-of-band management on a zone")
+ public OutOfBandManagementResponse enableDisableOutOfBandManagement(final DataCenter zone, final boolean enabled) {
+ dataCenterDetailsDao.persist(zone.getId(), OOBM_ENABLED_DETAIL, String.valueOf(enabled));
+ if (!enabled) {
+ transitionPowerStateToDisabled(hostDao.findByDataCenterId(zone.getId()));
+ }
+ return buildEnableDisableResponse(enabled);
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_ENABLEDISABLE, eventDescription = "enabling/disabling out-of-band management on a cluster")
+ public OutOfBandManagementResponse enableDisableOutOfBandManagement(final Cluster cluster, final boolean enabled) {
+ clusterDetailsDao.persist(cluster.getId(), OOBM_ENABLED_DETAIL, String.valueOf(enabled));
+ if (!enabled) {
+ transitionPowerStateToDisabled(hostDao.findByClusterId(cluster.getId()));
+ }
+ return buildEnableDisableResponse(enabled);
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_ENABLEDISABLE, eventDescription = "enabling/disabling out-of-band management on a host")
+ public OutOfBandManagementResponse enableDisableOutOfBandManagement(final Host host, final boolean enabled) {
+ final OutOfBandManagement outOfBandManagementConfig = outOfBandManagementDao.findByHost(host.getId());
+ if (outOfBandManagementConfig == null) {
+ final OutOfBandManagementResponse response = new OutOfBandManagementResponse(null);
+ response.setSuccess(false);
+ response.setResultDescription("Out-of-band management is not configured for the host. Please configure the host before enabling/disabling it.");
+ return response;
+ }
+ hostAlertCache.invalidate(host.getId());
+ outOfBandManagementConfig.setEnabled(enabled);
+ boolean updateResult = outOfBandManagementDao.update(outOfBandManagementConfig.getId(), (OutOfBandManagementVO) outOfBandManagementConfig);
+ if (updateResult && !enabled) {
+ transitionPowerStateToDisabled(Collections.singletonList(host));
+ }
+ return buildEnableDisableResponse(enabled && updateResult);
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_CONFIGURE, eventDescription = "updating out-of-band management configuration")
+ public OutOfBandManagementResponse configureOutOfBandManagement(final Host host, final ImmutableMap<OutOfBandManagement.Option, String> options) {
+ OutOfBandManagement outOfBandManagementConfig = outOfBandManagementDao.findByHost(host.getId());
+ if (outOfBandManagementConfig == null) {
+ outOfBandManagementConfig = outOfBandManagementDao.persist(new OutOfBandManagementVO(host.getId()));
+ }
+ outOfBandManagementConfig = updateConfig(outOfBandManagementConfig, options);
+ if (Strings.isNullOrEmpty(outOfBandManagementConfig.getDriver()) || !outOfBandManagementDriversMap.containsKey(outOfBandManagementConfig.getDriver().toLowerCase())) {
+ throw new CloudRuntimeException("Out-of-band management driver is not available. Please provide a valid driver name.");
+ }
+
+ boolean updatedConfig = outOfBandManagementDao.update(outOfBandManagementConfig.getId(), (OutOfBandManagementVO) outOfBandManagementConfig);
+ CallContext.current().setEventDetails("host id:" + host.getId() + " configuration:" + outOfBandManagementConfig.getAddress() + ":" + outOfBandManagementConfig.getPort());
+
+ if (!updatedConfig) {
+ throw new CloudRuntimeException("Failed to update out-of-band management config for the host in the database.");
+ }
+
+ String result = "Out-of-band management successfully configured for the host";
+ LOG.debug(result);
+
+ final OutOfBandManagementResponse response = new OutOfBandManagementResponse(outOfBandManagementDao.findByHost(host.getId()));
+ response.setResultDescription(result);
+ response.setSuccess(true);
+ return response;
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_ACTION, eventDescription = "issuing Host out-of-band management action", async = true)
+ public OutOfBandManagementResponse executeOutOfBandManagementPowerOperation(final Host host, final OutOfBandManagement.PowerOperation powerOperation, final Long timeout) {
+ checkOutOfBandManagementEnabledByZoneClusterHost(host);
+ final OutOfBandManagement outOfBandManagementConfig = outOfBandManagementDao.findByHost(host.getId());
+ final ImmutableMap<OutOfBandManagement.Option, String> options = getOptions(outOfBandManagementConfig);
+ final OutOfBandManagementDriver driver = getDriver(outOfBandManagementConfig);
+
+ Long actionTimeOut = timeout;
+ if (actionTimeOut == null) {
+ actionTimeOut = OutOfBandManagementActionTimeout.valueIn(host.getClusterId());
+ }
+
+ final OutOfBandManagementDriverPowerCommand cmd = new OutOfBandManagementDriverPowerCommand(options, actionTimeOut, powerOperation);
+ final OutOfBandManagementDriverResponse driverResponse = driver.execute(cmd);
+
+ if (driverResponse == null) {
+ throw new CloudRuntimeException(String.format("Out-of-band Management action (%s) on host (%s) failed due to no response from the driver", powerOperation, host.getUuid()));
+ }
+
+ if (powerOperation.equals(OutOfBandManagement.PowerOperation.STATUS)) {
+ transitionPowerState(driverResponse.toEvent(), outOfBandManagementConfig);
+ }
+
+ if (!driverResponse.isSuccess()) {
+ String errorMessage = String.format("Out-of-band Management action (%s) on host (%s) failed with error: %s", powerOperation, host.getUuid(), driverResponse.getError());
+ if (driverResponse.hasAuthFailure()) {
+ errorMessage = String.format("Out-of-band Management action (%s) on host (%s) failed due to authentication error: %s. Please check configured credentials.", powerOperation, host.getUuid(), driverResponse.getError());
+ sendAuthError(host, errorMessage);
+ }
+ if (!powerOperation.equals(OutOfBandManagement.PowerOperation.STATUS)) {
+ LOG.debug(errorMessage);
+ }
+ throw new CloudRuntimeException(errorMessage);
+ }
+
+ final OutOfBandManagementResponse response = new OutOfBandManagementResponse(outOfBandManagementDao.findByHost(host.getId()));
+ response.setSuccess(driverResponse.isSuccess());
+ response.setResultDescription(driverResponse.getResult());
+ response.setId(host.getUuid());
+ response.setOutOfBandManagementAction(powerOperation.toString());
+ return response;
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_CHANGE_PASSWORD, eventDescription = "updating out-of-band management password")
+ public OutOfBandManagementResponse changeOutOfBandManagementPassword(final Host host, final String newPassword) {
+ checkOutOfBandManagementEnabledByZoneClusterHost(host);
+ if (Strings.isNullOrEmpty(newPassword)) {
+ throw new CloudRuntimeException(String.format("Cannot change out-of-band management password as provided new-password is null or empty for the host %s.", host.getUuid()));
+ }
+ GlobalLock outOfBandManagementHostLock = GlobalLock.getInternLock(getOutOfBandManagementHostLock(host.getId()));
+ try {
+ if (outOfBandManagementHostLock.lock(ACQUIRE_GLOBAL_LOCK_TIMEOUT_FOR_HOST)) {
--- End diff --
we need to unlock too, so we're stuck with having critical code between if<get lock> <do stuff> <release lock> pattern
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r62322543
--- Diff: utils/src/main/java/org/apache/cloudstack/utils/process/ProcessRunner.java ---
@@ -0,0 +1,111 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+
+package org.apache.cloudstack.utils.process;
+
+import org.apache.log4j.Logger;
+
+import java.io.IOException;
+import java.util.List;
+
+public class ProcessRunner {
+ public static final Logger LOG = Logger.getLogger(ProcessRunner.class);
+
+ private String stdOutput;
+ private String stdError;
+ private int returnCode = -1;
--- End diff --
Consider making this class immutable and/or creating a static, immutable inner class (e.g. ``ProcessResult``).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61816897
--- Diff: test/integration/smoke/test_outofbandmanagement.py ---
@@ -0,0 +1,561 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import marvin
+from marvin.cloudstackTestCase import *
+from marvin.cloudstackAPI import *
+from marvin.lib.utils import *
+from marvin.lib.base import *
+from marvin.lib.common import *
+from marvin.lib.utils import (random_gen)
+from nose.plugins.attrib import attr
+
+from ipmisim.ipmisim import IpmiServerContext, IpmiServer, ThreadedIpmiServer
+
+import socket
+import sys
+import thread
+import time
+
+
+class TestOutOfBandManagement(cloudstackTestCase):
+ """ Test cases for out of band management
+ """
+
+ def setUp(self):
+ self.apiclient = self.testClient.getApiClient()
+ self.hypervisor = self.testClient.getHypervisorInfo()
+ self.dbclient = self.testClient.getDbConnection()
+ self.services = self.testClient.getParsedTestDataConfig()
+ self.mgtSvrDetails = self.config.__dict__["mgtSvr"][0].__dict__
+
+ self.zone = get_zone(self.apiclient, self.testClient.getZoneForTests())
+ self.host = None
+ self.server = None
+
+ # use random port for ipmisim
+ s = socket.socket()
+ s.bind(('', 0))
+ self.serverPort = s.getsockname()[1]
+ s.close()
+
+ self.cleanup = []
+
+
+ def tearDown(self):
+ try:
+ self.dbclient.execute("delete from oobm where port=%d" % self.getIpmiServerPort())
+ self.dbclient.execute("delete from mshost_peer where peer_runid=%s" % self.getFakeMsRunId())
+ self.dbclient.execute("delete from mshost where runid=%s" % self.getFakeMsRunId())
+ self.dbclient.execute("delete from cluster_details where name='outOfBandManagementEnabled'")
+ self.dbclient.execute("delete from data_center_details where name='outOfBandManagementEnabled'")
+ cleanup_resources(self.apiclient, self.cleanup)
+ if self.server:
+ self.server.shutdown()
+ self.server.server_close()
+ except Exception as e:
+ raise Exception("Warning: Exception during cleanup : %s" % e)
+
+
+ def getFakeMsId(self):
+ return 1234567890
+
+
+ def getFakeMsRunId(self):
+ return 123456
+
+
+ def getHost(self, hostId=None):
+ if self.host and hostId is None:
+ return self.host
+
+ response = list_hosts(
+ self.apiclient,
+ zoneid=self.zone.id,
+ type='Routing',
+ id=hostId
+ )
+ if len(response) > 0:
+ self.host = response[0]
+ return self.host
+ raise self.skipTest("No hosts found, skipping out-of-band management test")
+
+
+ def getIpmiServerIp(self):
+ s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
+ s.connect((self.mgtSvrDetails["mgtSvrIp"], self.mgtSvrDetails["port"]))
+ return s.getsockname()[0]
+
+
+ def getIpmiServerPort(self):
+ return self.serverPort
+
+
+ def getOobmConfigCmd(self):
+ cmd = configureOutOfBandManagement.configureOutOfBandManagementCmd()
+ cmd.driver = 'ipmitool' # The default available driver
+ cmd.address = self.getIpmiServerIp()
+ cmd.port = self.getIpmiServerPort()
+ cmd.username = 'admin'
+ cmd.password = 'password'
+ cmd.hostid = self.getHost().id
+ return cmd
+
+
+ def getOobmEnableCmd(self):
+ cmd = enableOutOfBandManagementForHost.enableOutOfBandManagementForHostCmd()
+ cmd.hostid = self.getHost().id
+ return cmd
+
+
+ def getOobmDisableCmd(self):
+ cmd = disableOutOfBandManagementForHost.disableOutOfBandManagementForHostCmd()
+ cmd.hostid = self.getHost().id
+ return cmd
+
+
+ def getOobmIssueActionCmd(self):
+ cmd = issueOutOfBandManagementPowerAction.issueOutOfBandManagementPowerActionCmd()
+ cmd.hostid = self.getHost().id
+ cmd.action = 'STATUS'
+ return cmd
+
+
+ def issuePowerActionCmd(self, action, timeout=None):
+ cmd = self.getOobmIssueActionCmd()
+ cmd.action = action
+ if timeout:
+ cmd.timeout = timeout
+ return self.apiclient.issueOutOfBandManagementPowerAction(cmd)
+
+
+ def configureAndEnableOobm(self):
+ self.apiclient.configureOutOfBandManagement(self.getOobmConfigCmd())
+ response = self.apiclient.enableOutOfBandManagementForHost(self.getOobmEnableCmd())
+ self.assertEqual(response.enabled, True)
+
+
+ def startIpmiServer(self):
+ def startIpmiServer(tname, server):
+ self.debug("Starting ipmisim server")
+ try:
+ server.serve_forever()
+ except Exception: pass
+ IpmiServerContext('reset')
+ ThreadedIpmiServer.allow_reuse_address = False
+ server = ThreadedIpmiServer(('0.0.0.0', self.getIpmiServerPort()), IpmiServer)
+ thread.start_new_thread(startIpmiServer, ("ipmi-server", server,))
+ self.server = server
+
+
+ def checkSyncToState(self, state, interval):
+ self.debug("Waiting for background thread to update powerstate to " + state)
+ time.sleep(1 + int(interval)*2/1000) # interval is in ms
+ response = self.getHost(hostId=self.getHost().id).outofbandmanagement
+ self.assertEqual(response.powerstate, state)
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_configure_invalid_driver(self):
+ """
+ Tests out-of-band management configuration with invalid driver
+ """
+ cmd = self.getOobmConfigCmd()
+ cmd.driver = 'randomDriverThatDoesNotExist'
+ try:
+ response = self.apiclient.configureOutOfBandManagement(cmd)
+ self.fail("Expected an exception to be thrown, failing")
+ except Exception: pass
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_configure_default_driver(self):
+ """
+ Tests out-of-band management configuration with valid data
+ """
+ cmd = self.getOobmConfigCmd()
+ response = self.apiclient.configureOutOfBandManagement(cmd)
+ self.assertEqual(response.hostid, cmd.hostid)
+ self.assertEqual(response.driver, cmd.driver)
+ self.assertEqual(response.address, cmd.address)
+ self.assertEqual(response.port, str(cmd.port))
+ self.assertEqual(response.username, cmd.username)
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_enable_feature_invalid(self):
+ """
+ Tests out-of-band management host enable feature with
+ invalid options
+ """
+ cmd = self.getOobmEnableCmd()
+ cmd.hostid = -1
+ try:
+ response = self.apiclient.enableOutOfBandManagementForHost(cmd)
+ self.fail("Expected an exception to be thrown, failing")
+ except Exception: pass
+
+ try:
+ cmd = enableOutOfBandManagementForCluster.enableOutOfBandManagementForClusterCmd()
+ response = self.apiclient.enableOutOfBandManagementForCluster(cmd)
+ self.fail("Expected an exception to be thrown, failing")
+ except Exception: pass
+
+ try:
+ cmd = enableOutOfBandManagementForZone.enableOutOfBandManagementForZoneCmd()
+ response = self.apiclient.enableOutOfBandManagementForZone(cmd)
+ self.fail("Expected an exception to be thrown, failing")
+ except Exception: pass
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_disable_feature_invalid(self):
+ """
+ Tests out-of-band management host disable feature with
+ invalid options
+ """
+ cmd = self.getOobmDisableCmd()
+ cmd.hostid = -1
+ try:
+ response = self.apiclient.disableOutOfBandManagementForHost(cmd)
+ self.fail("Expected an exception to be thrown, failing")
+ except Exception: pass
+
+ try:
+ cmd = disableOutOfBandManagementForCluster.disableOutOfBandManagementForClusterCmd()
+ response = self.apiclient.disableOutOfBandManagementForCluster(cmd)
+ self.fail("Expected an exception to be thrown, failing")
+ except Exception: pass
+
+ try:
+ cmd = disableOutOfBandManagementForZone.disableOutOfBandManagementForZoneCmd()
+ response = self.apiclient.disableOutOfBandManagementForZone(cmd)
+ self.fail("Expected an exception to be thrown, failing")
+ except Exception: pass
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_enable_feature_valid(self):
+ """
+ Tests out-of-band management host enable feature with
+ valid options
+ """
+ self.apiclient.configureOutOfBandManagement(self.getOobmConfigCmd())
+ cmd = self.getOobmEnableCmd()
+ response = self.apiclient.enableOutOfBandManagementForHost(cmd)
+ self.assertEqual(response.hostid, cmd.hostid)
+ self.assertEqual(response.enabled, True)
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_disable_feature_valid(self):
+ """
+ Tests out-of-band management host disable feature with
+ valid options
+ """
+
+ self.apiclient.configureOutOfBandManagement(self.getOobmConfigCmd())
+ cmd = self.getOobmDisableCmd()
+ response = self.apiclient.disableOutOfBandManagementForHost(cmd)
+ self.assertEqual(response.hostid, cmd.hostid)
+ self.assertEqual(response.enabled, False)
+
+ response = self.getHost(hostId=cmd.hostid).outofbandmanagement
+ self.assertEqual(response.powerstate, 'Disabled')
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_enabledisable_across_clusterzones(self):
+ """
+ Tests out-of-band management enable/disable feature at cluster
+ and zone level sequentially Zone > Cluster > Host
+ """
+ self.configureAndEnableOobm()
+ self.startIpmiServer()
+ bmc = IpmiServerContext().bmc
+ bmc.powerstate = 'off'
+
+ host = self.getHost()
+
+ # Disable at zone level
+ cmd = disableOutOfBandManagementForZone.disableOutOfBandManagementForZoneCmd()
+ cmd.zoneid = host.zoneid
+ response = self.apiclient.disableOutOfBandManagementForZone(cmd)
+
+ # Disable at cluster level
+ cmd = disableOutOfBandManagementForCluster.disableOutOfBandManagementForClusterCmd()
+ cmd.clusterid = host.clusterid
+ response = self.apiclient.disableOutOfBandManagementForCluster(cmd)
+
+ # Disable at host level
+ cmd = disableOutOfBandManagementForHost.disableOutOfBandManagementForHostCmd()
+ cmd.hostid = host.id
+ response = self.apiclient.disableOutOfBandManagementForHost(cmd)
+
+ try:
+ self.issuePowerActionCmd('STATUS')
+ self.fail("Exception was expected, oobm is disabled at zone level")
+ except Exception: pass
+
+ # Enable at zone level
+ cmd = enableOutOfBandManagementForZone.enableOutOfBandManagementForZoneCmd()
+ cmd.zoneid = host.zoneid
+ response = self.apiclient.enableOutOfBandManagementForZone(cmd)
+
+ try:
+ self.issuePowerActionCmd('STATUS')
+ self.fail("Exception was expected, oobm is disabled at cluster level")
+ except Exception: pass
+
+ # Check background thread syncs state to Disabled
+ response = self.getHost(hostId=host.id).outofbandmanagement
+ self.assertEqual(response.powerstate, 'Disabled')
+ self.dbclient.execute("update oobm set power_state='On' where port=%d" % self.getIpmiServerPort())
+ interval = list_configurations(
+ self.apiclient,
+ name='outofbandmanagement.sync.interval'
+ )[0].value
+ self.checkSyncToState('Disabled', interval)
+
+ # Enable at cluster level
+ cmd = enableOutOfBandManagementForCluster.enableOutOfBandManagementForClusterCmd()
+ cmd.clusterid = host.clusterid
+ response = self.apiclient.enableOutOfBandManagementForCluster(cmd)
+
+ try:
+ self.issuePowerActionCmd('STATUS')
+ self.fail("Exception was expected, oobm is disabled at host level")
+ except Exception: pass
+
+ # Enable at host level
+ cmd = enableOutOfBandManagementForHost.enableOutOfBandManagementForHostCmd()
+ cmd.hostid = host.id
+ response = self.apiclient.enableOutOfBandManagementForHost(cmd)
+
+ response = self.issuePowerActionCmd('STATUS')
+ self.assertEqual(response.powerstate, 'Off')
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_issue_power_status(self):
+ """
+ Tests out-of-band management issue power action
+ """
+ self.configureAndEnableOobm()
+ self.startIpmiServer()
+ bmc = IpmiServerContext().bmc
+ bmc.powerstate = 'on'
+ response = self.issuePowerActionCmd('STATUS')
+ self.assertEqual(response.powerstate, 'On')
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_issue_power_on(self):
+ """
+ Tests out-of-band management issue power on action
+ """
+ self.configureAndEnableOobm()
+ self.startIpmiServer()
+ self.issuePowerActionCmd('ON')
+ response = self.issuePowerActionCmd('STATUS')
+ self.assertEqual(response.powerstate, 'On')
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_issue_power_off(self):
+ """
+ Tests out-of-band management issue power off action
+ """
+ self.configureAndEnableOobm()
+ self.startIpmiServer()
+ self.issuePowerActionCmd('OFF')
+ response = self.issuePowerActionCmd('STATUS')
+ self.assertEqual(response.powerstate, 'Off')
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_issue_power_cycle(self):
+ """
+ Tests out-of-band management issue power cycle action
+ """
+ self.configureAndEnableOobm()
+ self.startIpmiServer()
+ self.issuePowerActionCmd('CYCLE')
+ response = self.issuePowerActionCmd('STATUS')
+ self.assertEqual(response.powerstate, 'On')
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_issue_power_reset(self):
+ """
+ Tests out-of-band management issue power reset action
+ """
+ self.configureAndEnableOobm()
+ self.startIpmiServer()
+ self.issuePowerActionCmd('RESET')
+ response = self.issuePowerActionCmd('STATUS')
+ self.assertEqual(response.powerstate, 'On')
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_issue_power_soft(self):
+ """
+ Tests out-of-band management issue power soft action
+ """
+ self.configureAndEnableOobm()
+ self.startIpmiServer()
+ self.issuePowerActionCmd('SOFT')
+ response = self.issuePowerActionCmd('STATUS')
+ self.assertEqual(response.powerstate, 'Off')
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_background_powerstate_sync(self):
+ """
+ Tests out-of-band management background powerstate sync
+ """
+ self.debug("Testing oobm background sync")
+ interval = list_configurations(
+ self.apiclient,
+ name='outofbandmanagement.sync.interval'
+ )[0].value
+
+ self.configureAndEnableOobm()
+ self.startIpmiServer()
+ bmc = IpmiServerContext().bmc
+
+ bmc.powerstate = 'on'
+ self.checkSyncToState('On', interval)
+
+ bmc.powerstate = 'off'
+ self.checkSyncToState('Off', interval)
+
+ self.server.shutdown()
+ self.server.server_close()
+
+ # Check for unknown state (ipmi server not reachable)
+ self.checkSyncToState('Unknown', interval)
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_multiple_mgmt_server_ownership(self):
+ """
+ Tests out-of-band management ownership expiry across multi-mgmt server
+ """
+ self.configureAndEnableOobm()
+
+ cloudstackVersion = Configurations.listCapabilities(self.apiclient).cloudstackversion
+
+ currentMsHosts = []
+ mshosts = self.dbclient.execute("select msid from mshost where version='%s' and removed is NULL and state='Up'" % (cloudstackVersion))
+ if len(mshosts) > 0:
+ currentMsHosts = map(lambda row: row[0], mshosts)
+
+ # Inject fake ms host
+ self.dbclient.execute("insert into mshost (msid,runid,name,state,version,service_ip,service_port,last_update) values (%s,%s,'oobm-marvin-fakebox', 'Down', '%s', '127.0.0.1', '22', NOW())" % (self.getFakeMsId(), self.getFakeMsRunId(), cloudstackVersion))
+
+ # Pass ownership to the fake ms id
+ self.dbclient.execute("update oobm set mgmt_server_id=%d where port=%d" % (self.getFakeMsId(), self.getIpmiServerPort()))
+
+ self.debug("Testing oobm background sync")
+ pingInterval = float(list_configurations(
+ self.apiclient,
+ name='ping.interval'
+ )[0].value)
+
+ pingTimeout = float(list_configurations(
+ self.apiclient,
+ name='ping.timeout'
+ )[0].value)
+
+ # Sleep until the ownership gets expired as the fake ms id is not reachable
+ for _ in range(10):
+ rows = self.dbclient.execute("select * from mshost_peer where peer_runid=%s" % self.getFakeMsRunId())
+ if len(rows) > 0:
+ self.debug("Mgmt server is now trying to contact the fake mgmt server")
+ self.dbclient.execute("update mshost set removed=now() where runid=%s" % self.getFakeMsRunId())
+ self.dbclient.execute("update mshost_peer set peer_state='Down' where peer_runid=%s" % self.getFakeMsRunId())
+ break
+ time.sleep(1 + (pingInterval * pingTimeout / 10))
+
+ for _ in range(100):
+ rows = self.dbclient.execute("select mgmt_server_id from oobm where port=%d" % (self.getIpmiServerPort()))
+ if len(rows) > 0 and rows[0][0] != self.getFakeMsId():
+ self.debug("Out-of-band management ownership expired as node was detected to be gone")
+ break
+ time.sleep(1 + (pingInterval * pingTimeout / 10))
--- End diff --
Consider using the ``wait_until`` function to collapse both of these wait loops.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61777144
--- Diff: api/src/org/apache/cloudstack/api/command/admin/outofbandmanagement/EnableOutOfBandManagementForClusterCmd.java ---
@@ -0,0 +1,105 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.api.command.admin.outofbandmanagement;
+
+import com.cloud.event.EventTypes;
+import com.cloud.exception.ConcurrentOperationException;
+import com.cloud.exception.InsufficientCapacityException;
+import com.cloud.exception.NetworkRuleConflictException;
+import com.cloud.exception.ResourceAllocationException;
+import com.cloud.exception.ResourceUnavailableException;
+import com.cloud.org.Cluster;
+import org.apache.cloudstack.acl.RoleType;
+import org.apache.cloudstack.api.APICommand;
+import org.apache.cloudstack.api.ApiConstants;
+import org.apache.cloudstack.api.ApiErrorCode;
+import org.apache.cloudstack.api.BaseAsyncCmd;
+import org.apache.cloudstack.api.BaseCmd;
+import org.apache.cloudstack.api.Parameter;
+import org.apache.cloudstack.api.ServerApiException;
+import org.apache.cloudstack.api.response.ClusterResponse;
+import org.apache.cloudstack.api.response.OutOfBandManagementResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagementService;
+
+import javax.inject.Inject;
+
+@APICommand(name = "enableOutOfBandManagementForCluster", description = "Enables out-of-band management for a cluster",
--- End diff --
Please add the version to this annotation
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61849798
--- Diff: test/integration/smoke/test_outofbandmanagement.py ---
@@ -0,0 +1,561 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import marvin
+from marvin.cloudstackTestCase import *
+from marvin.cloudstackAPI import *
+from marvin.lib.utils import *
+from marvin.lib.base import *
+from marvin.lib.common import *
+from marvin.lib.utils import (random_gen)
+from nose.plugins.attrib import attr
+
+from ipmisim.ipmisim import IpmiServerContext, IpmiServer, ThreadedIpmiServer
+
+import socket
+import sys
+import thread
+import time
+
+
+class TestOutOfBandManagement(cloudstackTestCase):
+ """ Test cases for out of band management
+ """
+
+ def setUp(self):
+ self.apiclient = self.testClient.getApiClient()
+ self.hypervisor = self.testClient.getHypervisorInfo()
+ self.dbclient = self.testClient.getDbConnection()
+ self.services = self.testClient.getParsedTestDataConfig()
+ self.mgtSvrDetails = self.config.__dict__["mgtSvr"][0].__dict__
+
+ self.zone = get_zone(self.apiclient, self.testClient.getZoneForTests())
+ self.host = None
+ self.server = None
+
+ # use random port for ipmisim
+ s = socket.socket()
+ s.bind(('', 0))
+ self.serverPort = s.getsockname()[1]
+ s.close()
+
+ self.cleanup = []
+
+
+ def tearDown(self):
+ try:
+ self.dbclient.execute("delete from oobm where port=%d" % self.getIpmiServerPort())
+ self.dbclient.execute("delete from mshost_peer where peer_runid=%s" % self.getFakeMsRunId())
+ self.dbclient.execute("delete from mshost where runid=%s" % self.getFakeMsRunId())
+ self.dbclient.execute("delete from cluster_details where name='outOfBandManagementEnabled'")
+ self.dbclient.execute("delete from data_center_details where name='outOfBandManagementEnabled'")
+ cleanup_resources(self.apiclient, self.cleanup)
+ if self.server:
+ self.server.shutdown()
+ self.server.server_close()
+ except Exception as e:
+ raise Exception("Warning: Exception during cleanup : %s" % e)
+
+
+ def getFakeMsId(self):
+ return 1234567890
+
+
+ def getFakeMsRunId(self):
+ return 123456
+
+
+ def getHost(self, hostId=None):
+ if self.host and hostId is None:
+ return self.host
+
+ response = list_hosts(
+ self.apiclient,
+ zoneid=self.zone.id,
+ type='Routing',
+ id=hostId
+ )
+ if len(response) > 0:
+ self.host = response[0]
+ return self.host
+ raise self.skipTest("No hosts found, skipping out-of-band management test")
+
+
+ def getIpmiServerIp(self):
+ s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
+ s.connect((self.mgtSvrDetails["mgtSvrIp"], self.mgtSvrDetails["port"]))
+ return s.getsockname()[0]
+
+
+ def getIpmiServerPort(self):
+ return self.serverPort
+
+
+ def getOobmConfigCmd(self):
+ cmd = configureOutOfBandManagement.configureOutOfBandManagementCmd()
+ cmd.driver = 'ipmitool' # The default available driver
+ cmd.address = self.getIpmiServerIp()
+ cmd.port = self.getIpmiServerPort()
+ cmd.username = 'admin'
+ cmd.password = 'password'
+ cmd.hostid = self.getHost().id
+ return cmd
+
+
+ def getOobmEnableCmd(self):
+ cmd = enableOutOfBandManagementForHost.enableOutOfBandManagementForHostCmd()
+ cmd.hostid = self.getHost().id
+ return cmd
+
+
+ def getOobmDisableCmd(self):
+ cmd = disableOutOfBandManagementForHost.disableOutOfBandManagementForHostCmd()
+ cmd.hostid = self.getHost().id
+ return cmd
+
+
+ def getOobmIssueActionCmd(self):
+ cmd = issueOutOfBandManagementPowerAction.issueOutOfBandManagementPowerActionCmd()
+ cmd.hostid = self.getHost().id
+ cmd.action = 'STATUS'
+ return cmd
+
+
+ def issuePowerActionCmd(self, action, timeout=None):
+ cmd = self.getOobmIssueActionCmd()
+ cmd.action = action
+ if timeout:
+ cmd.timeout = timeout
+ return self.apiclient.issueOutOfBandManagementPowerAction(cmd)
+
+
+ def configureAndEnableOobm(self):
+ self.apiclient.configureOutOfBandManagement(self.getOobmConfigCmd())
+ response = self.apiclient.enableOutOfBandManagementForHost(self.getOobmEnableCmd())
+ self.assertEqual(response.enabled, True)
+
+
+ def startIpmiServer(self):
+ def startIpmiServer(tname, server):
+ self.debug("Starting ipmisim server")
+ try:
+ server.serve_forever()
+ except Exception: pass
+ IpmiServerContext('reset')
+ ThreadedIpmiServer.allow_reuse_address = False
+ server = ThreadedIpmiServer(('0.0.0.0', self.getIpmiServerPort()), IpmiServer)
+ thread.start_new_thread(startIpmiServer, ("ipmi-server", server,))
+ self.server = server
+
+
+ def checkSyncToState(self, state, interval):
+ self.debug("Waiting for background thread to update powerstate to " + state)
+ time.sleep(1 + int(interval)*2/1000) # interval is in ms
+ response = self.getHost(hostId=self.getHost().id).outofbandmanagement
+ self.assertEqual(response.powerstate, state)
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_configure_invalid_driver(self):
+ """
+ Tests out-of-band management configuration with invalid driver
+ """
+ cmd = self.getOobmConfigCmd()
+ cmd.driver = 'randomDriverThatDoesNotExist'
+ try:
+ response = self.apiclient.configureOutOfBandManagement(cmd)
+ self.fail("Expected an exception to be thrown, failing")
+ except Exception: pass
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_configure_default_driver(self):
+ """
+ Tests out-of-band management configuration with valid data
+ """
+ cmd = self.getOobmConfigCmd()
+ response = self.apiclient.configureOutOfBandManagement(cmd)
+ self.assertEqual(response.hostid, cmd.hostid)
+ self.assertEqual(response.driver, cmd.driver)
+ self.assertEqual(response.address, cmd.address)
+ self.assertEqual(response.port, str(cmd.port))
+ self.assertEqual(response.username, cmd.username)
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_enable_feature_invalid(self):
+ """
+ Tests out-of-band management host enable feature with
+ invalid options
+ """
+ cmd = self.getOobmEnableCmd()
+ cmd.hostid = -1
+ try:
+ response = self.apiclient.enableOutOfBandManagementForHost(cmd)
+ self.fail("Expected an exception to be thrown, failing")
+ except Exception: pass
+
+ try:
+ cmd = enableOutOfBandManagementForCluster.enableOutOfBandManagementForClusterCmd()
+ response = self.apiclient.enableOutOfBandManagementForCluster(cmd)
+ self.fail("Expected an exception to be thrown, failing")
+ except Exception: pass
+
+ try:
+ cmd = enableOutOfBandManagementForZone.enableOutOfBandManagementForZoneCmd()
+ response = self.apiclient.enableOutOfBandManagementForZone(cmd)
+ self.fail("Expected an exception to be thrown, failing")
+ except Exception: pass
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_disable_feature_invalid(self):
+ """
+ Tests out-of-band management host disable feature with
+ invalid options
+ """
+ cmd = self.getOobmDisableCmd()
+ cmd.hostid = -1
+ try:
+ response = self.apiclient.disableOutOfBandManagementForHost(cmd)
+ self.fail("Expected an exception to be thrown, failing")
+ except Exception: pass
+
+ try:
+ cmd = disableOutOfBandManagementForCluster.disableOutOfBandManagementForClusterCmd()
+ response = self.apiclient.disableOutOfBandManagementForCluster(cmd)
+ self.fail("Expected an exception to be thrown, failing")
+ except Exception: pass
+
+ try:
+ cmd = disableOutOfBandManagementForZone.disableOutOfBandManagementForZoneCmd()
+ response = self.apiclient.disableOutOfBandManagementForZone(cmd)
+ self.fail("Expected an exception to be thrown, failing")
+ except Exception: pass
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_enable_feature_valid(self):
+ """
+ Tests out-of-band management host enable feature with
+ valid options
+ """
+ self.apiclient.configureOutOfBandManagement(self.getOobmConfigCmd())
+ cmd = self.getOobmEnableCmd()
+ response = self.apiclient.enableOutOfBandManagementForHost(cmd)
+ self.assertEqual(response.hostid, cmd.hostid)
+ self.assertEqual(response.enabled, True)
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_disable_feature_valid(self):
+ """
+ Tests out-of-band management host disable feature with
+ valid options
+ """
+
+ self.apiclient.configureOutOfBandManagement(self.getOobmConfigCmd())
+ cmd = self.getOobmDisableCmd()
+ response = self.apiclient.disableOutOfBandManagementForHost(cmd)
+ self.assertEqual(response.hostid, cmd.hostid)
+ self.assertEqual(response.enabled, False)
+
+ response = self.getHost(hostId=cmd.hostid).outofbandmanagement
+ self.assertEqual(response.powerstate, 'Disabled')
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_enabledisable_across_clusterzones(self):
+ """
+ Tests out-of-band management enable/disable feature at cluster
+ and zone level sequentially Zone > Cluster > Host
+ """
+ self.configureAndEnableOobm()
+ self.startIpmiServer()
+ bmc = IpmiServerContext().bmc
+ bmc.powerstate = 'off'
+
+ host = self.getHost()
+
+ # Disable at zone level
+ cmd = disableOutOfBandManagementForZone.disableOutOfBandManagementForZoneCmd()
+ cmd.zoneid = host.zoneid
+ response = self.apiclient.disableOutOfBandManagementForZone(cmd)
+
+ # Disable at cluster level
+ cmd = disableOutOfBandManagementForCluster.disableOutOfBandManagementForClusterCmd()
+ cmd.clusterid = host.clusterid
+ response = self.apiclient.disableOutOfBandManagementForCluster(cmd)
+
+ # Disable at host level
+ cmd = disableOutOfBandManagementForHost.disableOutOfBandManagementForHostCmd()
+ cmd.hostid = host.id
+ response = self.apiclient.disableOutOfBandManagementForHost(cmd)
+
+ try:
+ self.issuePowerActionCmd('STATUS')
+ self.fail("Exception was expected, oobm is disabled at zone level")
+ except Exception: pass
+
+ # Enable at zone level
+ cmd = enableOutOfBandManagementForZone.enableOutOfBandManagementForZoneCmd()
+ cmd.zoneid = host.zoneid
+ response = self.apiclient.enableOutOfBandManagementForZone(cmd)
+
+ try:
+ self.issuePowerActionCmd('STATUS')
+ self.fail("Exception was expected, oobm is disabled at cluster level")
+ except Exception: pass
+
+ # Check background thread syncs state to Disabled
+ response = self.getHost(hostId=host.id).outofbandmanagement
+ self.assertEqual(response.powerstate, 'Disabled')
+ self.dbclient.execute("update oobm set power_state='On' where port=%d" % self.getIpmiServerPort())
+ interval = list_configurations(
+ self.apiclient,
+ name='outofbandmanagement.sync.interval'
+ )[0].value
+ self.checkSyncToState('Disabled', interval)
+
+ # Enable at cluster level
+ cmd = enableOutOfBandManagementForCluster.enableOutOfBandManagementForClusterCmd()
+ cmd.clusterid = host.clusterid
+ response = self.apiclient.enableOutOfBandManagementForCluster(cmd)
+
+ try:
+ self.issuePowerActionCmd('STATUS')
+ self.fail("Exception was expected, oobm is disabled at host level")
+ except Exception: pass
+
+ # Enable at host level
+ cmd = enableOutOfBandManagementForHost.enableOutOfBandManagementForHostCmd()
+ cmd.hostid = host.id
+ response = self.apiclient.enableOutOfBandManagementForHost(cmd)
+
+ response = self.issuePowerActionCmd('STATUS')
+ self.assertEqual(response.powerstate, 'Off')
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_issue_power_status(self):
+ """
+ Tests out-of-band management issue power action
+ """
+ self.configureAndEnableOobm()
+ self.startIpmiServer()
+ bmc = IpmiServerContext().bmc
+ bmc.powerstate = 'on'
+ response = self.issuePowerActionCmd('STATUS')
+ self.assertEqual(response.powerstate, 'On')
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_issue_power_on(self):
+ """
+ Tests out-of-band management issue power on action
+ """
+ self.configureAndEnableOobm()
+ self.startIpmiServer()
+ self.issuePowerActionCmd('ON')
+ response = self.issuePowerActionCmd('STATUS')
+ self.assertEqual(response.powerstate, 'On')
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_issue_power_off(self):
+ """
+ Tests out-of-band management issue power off action
+ """
+ self.configureAndEnableOobm()
+ self.startIpmiServer()
+ self.issuePowerActionCmd('OFF')
+ response = self.issuePowerActionCmd('STATUS')
+ self.assertEqual(response.powerstate, 'Off')
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_issue_power_cycle(self):
+ """
+ Tests out-of-band management issue power cycle action
+ """
+ self.configureAndEnableOobm()
+ self.startIpmiServer()
+ self.issuePowerActionCmd('CYCLE')
+ response = self.issuePowerActionCmd('STATUS')
+ self.assertEqual(response.powerstate, 'On')
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_issue_power_reset(self):
+ """
+ Tests out-of-band management issue power reset action
+ """
+ self.configureAndEnableOobm()
+ self.startIpmiServer()
+ self.issuePowerActionCmd('RESET')
+ response = self.issuePowerActionCmd('STATUS')
+ self.assertEqual(response.powerstate, 'On')
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_issue_power_soft(self):
+ """
+ Tests out-of-band management issue power soft action
+ """
+ self.configureAndEnableOobm()
+ self.startIpmiServer()
+ self.issuePowerActionCmd('SOFT')
+ response = self.issuePowerActionCmd('STATUS')
+ self.assertEqual(response.powerstate, 'Off')
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_background_powerstate_sync(self):
+ """
+ Tests out-of-band management background powerstate sync
+ """
+ self.debug("Testing oobm background sync")
+ interval = list_configurations(
+ self.apiclient,
+ name='outofbandmanagement.sync.interval'
+ )[0].value
+
+ self.configureAndEnableOobm()
+ self.startIpmiServer()
+ bmc = IpmiServerContext().bmc
+
+ bmc.powerstate = 'on'
+ self.checkSyncToState('On', interval)
+
+ bmc.powerstate = 'off'
+ self.checkSyncToState('Off', interval)
+
+ self.server.shutdown()
+ self.server.server_close()
+
+ # Check for unknown state (ipmi server not reachable)
+ self.checkSyncToState('Unknown', interval)
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_multiple_mgmt_server_ownership(self):
--- End diff --
This is what this test does, by simulating another mgmt server by doing a direct db manipulation and checks how the current mgmgt server behaves.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61795721
--- Diff: server/src/org/apache/cloudstack/outofbandmanagement/OutOfBandManagementServiceImpl.java ---
@@ -0,0 +1,532 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.outofbandmanagement;
+
+import com.cloud.alert.AlertManager;
+import com.cloud.dc.ClusterDetailsDao;
+import com.cloud.dc.ClusterDetailsVO;
+import com.cloud.dc.DataCenter;
+import com.cloud.dc.DataCenterDetailVO;
+import com.cloud.dc.dao.DataCenterDetailsDao;
+import com.cloud.domain.Domain;
+import com.cloud.event.ActionEvent;
+import com.cloud.event.ActionEventUtils;
+import com.cloud.event.EventTypes;
+import com.cloud.host.Host;
+import com.cloud.host.dao.HostDao;
+import com.cloud.org.Cluster;
+import com.cloud.utils.component.Manager;
+import com.cloud.utils.component.ManagerBase;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.db.Transaction;
+import com.cloud.utils.db.TransactionCallback;
+import com.cloud.utils.db.TransactionStatus;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.fsm.NoTransitionException;
+import com.google.common.base.Strings;
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import com.google.common.collect.ImmutableMap;
+import org.apache.cloudstack.api.response.OutOfBandManagementResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.framework.config.ConfigKey;
+import org.apache.cloudstack.framework.config.Configurable;
+import org.apache.cloudstack.outofbandmanagement.dao.OutOfBandManagementDao;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverChangePasswordCommand;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverPowerCommand;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverResponse;
+import org.apache.cloudstack.utils.identity.ManagementServerNode;
+import org.apache.log4j.Logger;
+import org.springframework.stereotype.Component;
+
+import javax.ejb.Local;
+import javax.inject.Inject;
+import javax.naming.ConfigurationException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+@Component
+@Local(value = {OutOfBandManagementService.class})
--- End diff --
``@Local`` annotations are no longer required and can be safely removed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61849729
--- Diff: test/integration/smoke/test_outofbandmanagement.py ---
@@ -0,0 +1,561 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import marvin
+from marvin.cloudstackTestCase import *
+from marvin.cloudstackAPI import *
+from marvin.lib.utils import *
+from marvin.lib.base import *
+from marvin.lib.common import *
+from marvin.lib.utils import (random_gen)
+from nose.plugins.attrib import attr
+
+from ipmisim.ipmisim import IpmiServerContext, IpmiServer, ThreadedIpmiServer
+
+import socket
+import sys
+import thread
+import time
+
+
+class TestOutOfBandManagement(cloudstackTestCase):
+ """ Test cases for out of band management
+ """
+
+ def setUp(self):
+ self.apiclient = self.testClient.getApiClient()
+ self.hypervisor = self.testClient.getHypervisorInfo()
+ self.dbclient = self.testClient.getDbConnection()
+ self.services = self.testClient.getParsedTestDataConfig()
+ self.mgtSvrDetails = self.config.__dict__["mgtSvr"][0].__dict__
+
+ self.zone = get_zone(self.apiclient, self.testClient.getZoneForTests())
+ self.host = None
+ self.server = None
+
+ # use random port for ipmisim
+ s = socket.socket()
+ s.bind(('', 0))
+ self.serverPort = s.getsockname()[1]
+ s.close()
+
+ self.cleanup = []
+
+
+ def tearDown(self):
+ try:
+ self.dbclient.execute("delete from oobm where port=%d" % self.getIpmiServerPort())
+ self.dbclient.execute("delete from mshost_peer where peer_runid=%s" % self.getFakeMsRunId())
+ self.dbclient.execute("delete from mshost where runid=%s" % self.getFakeMsRunId())
+ self.dbclient.execute("delete from cluster_details where name='outOfBandManagementEnabled'")
+ self.dbclient.execute("delete from data_center_details where name='outOfBandManagementEnabled'")
+ cleanup_resources(self.apiclient, self.cleanup)
+ if self.server:
+ self.server.shutdown()
+ self.server.server_close()
+ except Exception as e:
+ raise Exception("Warning: Exception during cleanup : %s" % e)
+
+
+ def getFakeMsId(self):
+ return 1234567890
+
+
+ def getFakeMsRunId(self):
+ return 123456
+
+
+ def getHost(self, hostId=None):
+ if self.host and hostId is None:
+ return self.host
+
+ response = list_hosts(
+ self.apiclient,
+ zoneid=self.zone.id,
+ type='Routing',
+ id=hostId
+ )
+ if len(response) > 0:
+ self.host = response[0]
+ return self.host
+ raise self.skipTest("No hosts found, skipping out-of-band management test")
+
+
+ def getIpmiServerIp(self):
+ s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
+ s.connect((self.mgtSvrDetails["mgtSvrIp"], self.mgtSvrDetails["port"]))
+ return s.getsockname()[0]
+
+
+ def getIpmiServerPort(self):
+ return self.serverPort
+
+
+ def getOobmConfigCmd(self):
+ cmd = configureOutOfBandManagement.configureOutOfBandManagementCmd()
+ cmd.driver = 'ipmitool' # The default available driver
+ cmd.address = self.getIpmiServerIp()
+ cmd.port = self.getIpmiServerPort()
+ cmd.username = 'admin'
+ cmd.password = 'password'
+ cmd.hostid = self.getHost().id
+ return cmd
+
+
+ def getOobmEnableCmd(self):
+ cmd = enableOutOfBandManagementForHost.enableOutOfBandManagementForHostCmd()
+ cmd.hostid = self.getHost().id
+ return cmd
+
+
+ def getOobmDisableCmd(self):
+ cmd = disableOutOfBandManagementForHost.disableOutOfBandManagementForHostCmd()
+ cmd.hostid = self.getHost().id
+ return cmd
+
+
+ def getOobmIssueActionCmd(self):
+ cmd = issueOutOfBandManagementPowerAction.issueOutOfBandManagementPowerActionCmd()
+ cmd.hostid = self.getHost().id
+ cmd.action = 'STATUS'
+ return cmd
+
+
+ def issuePowerActionCmd(self, action, timeout=None):
+ cmd = self.getOobmIssueActionCmd()
+ cmd.action = action
+ if timeout:
+ cmd.timeout = timeout
+ return self.apiclient.issueOutOfBandManagementPowerAction(cmd)
+
+
+ def configureAndEnableOobm(self):
+ self.apiclient.configureOutOfBandManagement(self.getOobmConfigCmd())
+ response = self.apiclient.enableOutOfBandManagementForHost(self.getOobmEnableCmd())
+ self.assertEqual(response.enabled, True)
+
+
+ def startIpmiServer(self):
+ def startIpmiServer(tname, server):
+ self.debug("Starting ipmisim server")
+ try:
+ server.serve_forever()
+ except Exception: pass
+ IpmiServerContext('reset')
+ ThreadedIpmiServer.allow_reuse_address = False
+ server = ThreadedIpmiServer(('0.0.0.0', self.getIpmiServerPort()), IpmiServer)
+ thread.start_new_thread(startIpmiServer, ("ipmi-server", server,))
+ self.server = server
+
+
+ def checkSyncToState(self, state, interval):
+ self.debug("Waiting for background thread to update powerstate to " + state)
+ time.sleep(1 + int(interval)*2/1000) # interval is in ms
+ response = self.getHost(hostId=self.getHost().id).outofbandmanagement
+ self.assertEqual(response.powerstate, state)
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_configure_invalid_driver(self):
+ """
+ Tests out-of-band management configuration with invalid driver
+ """
+ cmd = self.getOobmConfigCmd()
+ cmd.driver = 'randomDriverThatDoesNotExist'
+ try:
+ response = self.apiclient.configureOutOfBandManagement(cmd)
+ self.fail("Expected an exception to be thrown, failing")
+ except Exception: pass
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_configure_default_driver(self):
+ """
+ Tests out-of-band management configuration with valid data
+ """
+ cmd = self.getOobmConfigCmd()
+ response = self.apiclient.configureOutOfBandManagement(cmd)
+ self.assertEqual(response.hostid, cmd.hostid)
+ self.assertEqual(response.driver, cmd.driver)
+ self.assertEqual(response.address, cmd.address)
+ self.assertEqual(response.port, str(cmd.port))
+ self.assertEqual(response.username, cmd.username)
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_enable_feature_invalid(self):
+ """
+ Tests out-of-band management host enable feature with
+ invalid options
+ """
+ cmd = self.getOobmEnableCmd()
+ cmd.hostid = -1
+ try:
+ response = self.apiclient.enableOutOfBandManagementForHost(cmd)
+ self.fail("Expected an exception to be thrown, failing")
+ except Exception: pass
+
+ try:
+ cmd = enableOutOfBandManagementForCluster.enableOutOfBandManagementForClusterCmd()
+ response = self.apiclient.enableOutOfBandManagementForCluster(cmd)
+ self.fail("Expected an exception to be thrown, failing")
+ except Exception: pass
+
+ try:
+ cmd = enableOutOfBandManagementForZone.enableOutOfBandManagementForZoneCmd()
+ response = self.apiclient.enableOutOfBandManagementForZone(cmd)
+ self.fail("Expected an exception to be thrown, failing")
+ except Exception: pass
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_disable_feature_invalid(self):
+ """
+ Tests out-of-band management host disable feature with
+ invalid options
+ """
+ cmd = self.getOobmDisableCmd()
+ cmd.hostid = -1
+ try:
+ response = self.apiclient.disableOutOfBandManagementForHost(cmd)
+ self.fail("Expected an exception to be thrown, failing")
+ except Exception: pass
+
+ try:
+ cmd = disableOutOfBandManagementForCluster.disableOutOfBandManagementForClusterCmd()
+ response = self.apiclient.disableOutOfBandManagementForCluster(cmd)
+ self.fail("Expected an exception to be thrown, failing")
+ except Exception: pass
+
+ try:
+ cmd = disableOutOfBandManagementForZone.disableOutOfBandManagementForZoneCmd()
+ response = self.apiclient.disableOutOfBandManagementForZone(cmd)
+ self.fail("Expected an exception to be thrown, failing")
+ except Exception: pass
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_enable_feature_valid(self):
+ """
+ Tests out-of-band management host enable feature with
+ valid options
+ """
+ self.apiclient.configureOutOfBandManagement(self.getOobmConfigCmd())
+ cmd = self.getOobmEnableCmd()
+ response = self.apiclient.enableOutOfBandManagementForHost(cmd)
+ self.assertEqual(response.hostid, cmd.hostid)
+ self.assertEqual(response.enabled, True)
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_disable_feature_valid(self):
+ """
+ Tests out-of-band management host disable feature with
+ valid options
+ """
+
+ self.apiclient.configureOutOfBandManagement(self.getOobmConfigCmd())
+ cmd = self.getOobmDisableCmd()
+ response = self.apiclient.disableOutOfBandManagementForHost(cmd)
+ self.assertEqual(response.hostid, cmd.hostid)
+ self.assertEqual(response.enabled, False)
+
+ response = self.getHost(hostId=cmd.hostid).outofbandmanagement
+ self.assertEqual(response.powerstate, 'Disabled')
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_enabledisable_across_clusterzones(self):
+ """
+ Tests out-of-band management enable/disable feature at cluster
+ and zone level sequentially Zone > Cluster > Host
+ """
+ self.configureAndEnableOobm()
+ self.startIpmiServer()
+ bmc = IpmiServerContext().bmc
+ bmc.powerstate = 'off'
+
+ host = self.getHost()
+
+ # Disable at zone level
+ cmd = disableOutOfBandManagementForZone.disableOutOfBandManagementForZoneCmd()
+ cmd.zoneid = host.zoneid
+ response = self.apiclient.disableOutOfBandManagementForZone(cmd)
+
+ # Disable at cluster level
+ cmd = disableOutOfBandManagementForCluster.disableOutOfBandManagementForClusterCmd()
+ cmd.clusterid = host.clusterid
+ response = self.apiclient.disableOutOfBandManagementForCluster(cmd)
+
+ # Disable at host level
+ cmd = disableOutOfBandManagementForHost.disableOutOfBandManagementForHostCmd()
+ cmd.hostid = host.id
+ response = self.apiclient.disableOutOfBandManagementForHost(cmd)
+
+ try:
+ self.issuePowerActionCmd('STATUS')
+ self.fail("Exception was expected, oobm is disabled at zone level")
+ except Exception: pass
+
+ # Enable at zone level
+ cmd = enableOutOfBandManagementForZone.enableOutOfBandManagementForZoneCmd()
+ cmd.zoneid = host.zoneid
+ response = self.apiclient.enableOutOfBandManagementForZone(cmd)
+
+ try:
+ self.issuePowerActionCmd('STATUS')
+ self.fail("Exception was expected, oobm is disabled at cluster level")
+ except Exception: pass
+
+ # Check background thread syncs state to Disabled
+ response = self.getHost(hostId=host.id).outofbandmanagement
+ self.assertEqual(response.powerstate, 'Disabled')
+ self.dbclient.execute("update oobm set power_state='On' where port=%d" % self.getIpmiServerPort())
+ interval = list_configurations(
+ self.apiclient,
+ name='outofbandmanagement.sync.interval'
+ )[0].value
+ self.checkSyncToState('Disabled', interval)
+
+ # Enable at cluster level
+ cmd = enableOutOfBandManagementForCluster.enableOutOfBandManagementForClusterCmd()
+ cmd.clusterid = host.clusterid
+ response = self.apiclient.enableOutOfBandManagementForCluster(cmd)
+
+ try:
+ self.issuePowerActionCmd('STATUS')
+ self.fail("Exception was expected, oobm is disabled at host level")
+ except Exception: pass
+
+ # Enable at host level
+ cmd = enableOutOfBandManagementForHost.enableOutOfBandManagementForHostCmd()
+ cmd.hostid = host.id
+ response = self.apiclient.enableOutOfBandManagementForHost(cmd)
+
+ response = self.issuePowerActionCmd('STATUS')
+ self.assertEqual(response.powerstate, 'Off')
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_issue_power_status(self):
+ """
+ Tests out-of-band management issue power action
+ """
+ self.configureAndEnableOobm()
+ self.startIpmiServer()
+ bmc = IpmiServerContext().bmc
+ bmc.powerstate = 'on'
+ response = self.issuePowerActionCmd('STATUS')
+ self.assertEqual(response.powerstate, 'On')
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_issue_power_on(self):
+ """
+ Tests out-of-band management issue power on action
+ """
+ self.configureAndEnableOobm()
+ self.startIpmiServer()
+ self.issuePowerActionCmd('ON')
+ response = self.issuePowerActionCmd('STATUS')
+ self.assertEqual(response.powerstate, 'On')
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_issue_power_off(self):
+ """
+ Tests out-of-band management issue power off action
+ """
+ self.configureAndEnableOobm()
+ self.startIpmiServer()
+ self.issuePowerActionCmd('OFF')
+ response = self.issuePowerActionCmd('STATUS')
+ self.assertEqual(response.powerstate, 'Off')
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_issue_power_cycle(self):
+ """
+ Tests out-of-band management issue power cycle action
+ """
+ self.configureAndEnableOobm()
+ self.startIpmiServer()
+ self.issuePowerActionCmd('CYCLE')
+ response = self.issuePowerActionCmd('STATUS')
+ self.assertEqual(response.powerstate, 'On')
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_issue_power_reset(self):
+ """
+ Tests out-of-band management issue power reset action
+ """
+ self.configureAndEnableOobm()
+ self.startIpmiServer()
+ self.issuePowerActionCmd('RESET')
+ response = self.issuePowerActionCmd('STATUS')
+ self.assertEqual(response.powerstate, 'On')
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_issue_power_soft(self):
+ """
+ Tests out-of-band management issue power soft action
+ """
+ self.configureAndEnableOobm()
+ self.startIpmiServer()
+ self.issuePowerActionCmd('SOFT')
+ response = self.issuePowerActionCmd('STATUS')
+ self.assertEqual(response.powerstate, 'Off')
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_background_powerstate_sync(self):
+ """
+ Tests out-of-band management background powerstate sync
+ """
+ self.debug("Testing oobm background sync")
+ interval = list_configurations(
+ self.apiclient,
+ name='outofbandmanagement.sync.interval'
+ )[0].value
+
+ self.configureAndEnableOobm()
+ self.startIpmiServer()
+ bmc = IpmiServerContext().bmc
+
+ bmc.powerstate = 'on'
+ self.checkSyncToState('On', interval)
+
+ bmc.powerstate = 'off'
+ self.checkSyncToState('Off', interval)
+
+ self.server.shutdown()
+ self.server.server_close()
+
+ # Check for unknown state (ipmi server not reachable)
+ self.checkSyncToState('Unknown', interval)
+
+
+ @attr(tags=["smoke", "advanced"])
+ def test_oobm_multiple_mgmt_server_ownership(self):
+ """
+ Tests out-of-band management ownership expiry across multi-mgmt server
+ """
+ self.configureAndEnableOobm()
+
+ cloudstackVersion = Configurations.listCapabilities(self.apiclient).cloudstackversion
+
+ currentMsHosts = []
+ mshosts = self.dbclient.execute("select msid from mshost where version='%s' and removed is NULL and state='Up'" % (cloudstackVersion))
+ if len(mshosts) > 0:
+ currentMsHosts = map(lambda row: row[0], mshosts)
+
+ # Inject fake ms host
+ self.dbclient.execute("insert into mshost (msid,runid,name,state,version,service_ip,service_port,last_update) values (%s,%s,'oobm-marvin-fakebox', 'Down', '%s', '127.0.0.1', '22', NOW())" % (self.getFakeMsId(), self.getFakeMsRunId(), cloudstackVersion))
+
+ # Pass ownership to the fake ms id
+ self.dbclient.execute("update oobm set mgmt_server_id=%d where port=%d" % (self.getFakeMsId(), self.getIpmiServerPort()))
+
+ self.debug("Testing oobm background sync")
+ pingInterval = float(list_configurations(
+ self.apiclient,
+ name='ping.interval'
+ )[0].value)
+
+ pingTimeout = float(list_configurations(
+ self.apiclient,
+ name='ping.timeout'
+ )[0].value)
+
+ # Sleep until the ownership gets expired as the fake ms id is not reachable
+ for _ in range(10):
+ rows = self.dbclient.execute("select * from mshost_peer where peer_runid=%s" % self.getFakeMsRunId())
+ if len(rows) > 0:
+ self.debug("Mgmt server is now trying to contact the fake mgmt server")
+ self.dbclient.execute("update mshost set removed=now() where runid=%s" % self.getFakeMsRunId())
+ self.dbclient.execute("update mshost_peer set peer_state='Down' where peer_runid=%s" % self.getFakeMsRunId())
+ break
+ time.sleep(1 + (pingInterval * pingTimeout / 10))
+
+ for _ in range(100):
+ rows = self.dbclient.execute("select mgmt_server_id from oobm where port=%d" % (self.getIpmiServerPort()))
+ if len(rows) > 0 and rows[0][0] != self.getFakeMsId():
+ self.debug("Out-of-band management ownership expired as node was detected to be gone")
+ break
+ time.sleep(1 + (pingInterval * pingTimeout / 10))
--- End diff --
fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r60885329
--- Diff: api/src/org/apache/cloudstack/api/command/admin/outofbandmanagement/ConfigureOutOfBandManagementCmd.java ---
@@ -0,0 +1,127 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.api.command.admin.outofbandmanagement;
+
+import com.cloud.exception.ConcurrentOperationException;
+import com.cloud.exception.InsufficientCapacityException;
+import com.cloud.exception.NetworkRuleConflictException;
+import com.cloud.exception.ResourceAllocationException;
+import com.cloud.exception.ResourceUnavailableException;
+import com.cloud.host.Host;
+import com.google.common.base.Strings;
+import com.google.common.collect.ImmutableMap;
+import org.apache.cloudstack.acl.RoleType;
+import org.apache.cloudstack.api.APICommand;
+import org.apache.cloudstack.api.ApiConstants;
+import org.apache.cloudstack.api.ApiErrorCode;
+import org.apache.cloudstack.api.BaseCmd;
+import org.apache.cloudstack.api.Parameter;
+import org.apache.cloudstack.api.ServerApiException;
+import org.apache.cloudstack.api.response.HostResponse;
+import org.apache.cloudstack.api.response.OutOfBandManagementResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagement;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagementService;
+
+import javax.inject.Inject;
+
+@APICommand(name = "configureOutOfBandManagement", description = "Configures a host's out-of-band management interface",
+ responseObject = OutOfBandManagementResponse.class, requestHasSensitiveInfo = true, responseHasSensitiveInfo = false, authorized = {RoleType.Admin})
+public class ConfigureOutOfBandManagementCmd extends BaseCmd {
+ @Inject
+ private OutOfBandManagementService outOfBandManagementService;
+
+ /////////////////////////////////////////////////////
+ //////////////// API parameters /////////////////////
+ /////////////////////////////////////////////////////
+
+ @Parameter(name = ApiConstants.HOST_ID, type = CommandType.UUID, entityType = HostResponse.class, required = true, description = "the ID of the host")
+ private Long hostId;
+
+ @Parameter(name = ApiConstants.DRIVER, type = CommandType.STRING, required = true, description = "the host management interface driver, for example: ipmitool")
+ private String driver;
+
+ @Parameter(name = ApiConstants.IP_ADDRESS, type = CommandType.STRING, required = true, description = "the host management interface IP address")
--- End diff --
@wido thanks, I'll update the arg name to 'address' instead of 'ipaddress' as we also may want to use domain names here. It's left to interpretation of how a out-of-band management plugin would execute requests with these configs; currently the default ipmitool plugin forks out a process with arguments, so ipv6/ipv4/domain names are supported as long as ipmitool process is able to use them and execute a request.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by swill <gi...@git.apache.org>.
Github user swill commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-218861696
I see the following error which is causing the PR #1297 to fail. Suggestions?
```
+---------------------------------------------+----------------------+--------+
| test_oobm_zchange_password | exceptions.Exception | 5.220 |
+---------------------------------------------+----------------------+--------+
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r62322259
--- Diff: server/test/com/cloud/resource/MockResourceManagerImpl.java ---
@@ -172,6 +173,12 @@ public Cluster getCluster(final Long clusterId) {
return null;
}
+ @Override
+ public DataCenter getZone(Long zoneId) {
+ // TODO Auto-generated method stub
--- End diff --
Because we have tools to search for TODOs, please remove the comment if there is nothing outstanding to be done.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-218059135
@jburwell as mentioned in comments above, due to a bug in ipmitool itself (https://bugzilla.redhat.com/show_bug.cgi?id=1286035) some executions may fail. I'll add code to skip/workaround that known bug for the specific failing test, I had already added some code to skip test when the specific ipmitool bug was hit.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r62612254
--- Diff: utils/src/main/java/org/apache/cloudstack/utils/process/ProcessResult.java ---
@@ -0,0 +1,46 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.utils.process;
+
+public class ProcessResult {
--- End diff --
Will fix that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r62661197
--- Diff: utils/src/main/java/org/apache/cloudstack/utils/process/ProcessRunner.java ---
@@ -0,0 +1,112 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+
+package org.apache.cloudstack.utils.process;
+
+import com.cloud.utils.concurrency.NamedThreadFactory;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Strings;
+import org.apache.log4j.Logger;
+import org.joda.time.Duration;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.List;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+public class ProcessRunner {
+ public static final Logger LOG = Logger.getLogger(ProcessRunner.class);
+
+ private static final ExecutorService processExecutor = Executors.newCachedThreadPool(new NamedThreadFactory("ProcessRunner"));
+
+ private static String readStream(final InputStream inputStream) {
+ String text = null;
+ try {
+ final BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(inputStream));
+ String line;
+ while ((line = bufferedReader.readLine()) != null) {
+ if (Strings.isNullOrEmpty(text)) {
+ text = line;
+ } else {
+ text = text + "\n" + line;
+ }
+ }
+ } catch (IOException e) {
+ if (LOG.isTraceEnabled()) {
+ LOG.trace("ProcessRunner::readStream failed due to: " + e.getMessage());
+ }
+ }
+ return text;
+ }
+
+ public static ProcessResult executeCommands(final List<String> commands, final Duration timeOut) {
+ Preconditions.checkArgument(commands != null && timeOut != null);
+
+ int retVal = -2;
+ String stdOutput = null;
+ String stdError = null;
+
+ try {
+ final Process process = new ProcessBuilder().command(commands).start();
+ if (timeOut.getStandardSeconds() > 0) {
--- End diff --
@rhtyd I don't think it acceptable to allow a process without any time boundary. The concern is that a series of stalled commands could consume the thread pool and starve out other clients. However, as a convenience to users of the class, it also makes senses that we would require the specification of a timeout on each call.
A solution to this issue would be as follows:
1. Define a ``private final static Duration DEFAULT_COMMAND_TIMEOUT``
1. Provide an ``executeCommands`` override implementation:
```
public static ProcessResult executeCommands(final List<String> commands) {
return executeCommands(commands, DEFAULT_COMMAND_TIMEOUT);
}
```
1. Modify the preconditions of this method to require a timeout greater than zero and simplify it to remove handling of that case.
This approach not only ensures that all commands are (softly) time-bounded, but it also avoid the [flag argument anti-pattern](http://martinfowler.com/bliki/FlagArgument.html).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61845126
--- Diff: engine/schema/src/org/apache/cloudstack/outofbandmanagement/dao/OutOfBandManagementDaoImpl.java ---
@@ -0,0 +1,163 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.outofbandmanagement.dao;
+
+import com.cloud.utils.DateUtil;
+import com.cloud.utils.db.Attribute;
+import com.cloud.utils.db.DB;
+import com.cloud.utils.db.Filter;
+import com.cloud.utils.db.GenericDaoBase;
+import com.cloud.utils.db.SearchBuilder;
+import com.cloud.utils.db.SearchCriteria;
+import com.cloud.utils.db.TransactionLegacy;
+import com.cloud.utils.db.UpdateBuilder;
+import com.cloud.utils.exception.CloudRuntimeException;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagement;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagementVO;
+import org.apache.log4j.Logger;
+import org.springframework.stereotype.Component;
+
+import javax.ejb.Local;
+import java.sql.PreparedStatement;
+import java.sql.SQLException;
+import java.util.List;
+
+@DB
+@Component
+@Local(value = {OutOfBandManagementDao.class})
+public class OutOfBandManagementDaoImpl extends GenericDaoBase<OutOfBandManagementVO, Long> implements OutOfBandManagementDao {
+ private static final Logger LOG = Logger.getLogger(OutOfBandManagementDaoImpl.class);
+
+ private SearchBuilder<OutOfBandManagementVO> HostSearch;
+ private SearchBuilder<OutOfBandManagementVO> ManagementServerSearch;
+ private SearchBuilder<OutOfBandManagementVO> OutOfBandManagementOwnerSearch;
+ private SearchBuilder<OutOfBandManagementVO> StateUpdateSearch;
+
+ private Attribute PowerStateAttr;
+ private Attribute MsIdAttr;
+ private Attribute UpdateTimeAttr;
+
+ public OutOfBandManagementDaoImpl() {
+ super();
+
+ HostSearch = createSearchBuilder();
+ HostSearch.and("hostId", HostSearch.entity().getHostId(), SearchCriteria.Op.EQ);
+ HostSearch.done();
+
+ ManagementServerSearch = createSearchBuilder();
+ ManagementServerSearch.and("server", ManagementServerSearch.entity().getManagementServerId(), SearchCriteria.Op.EQ);
+ ManagementServerSearch.done();
+
+ OutOfBandManagementOwnerSearch = createSearchBuilder();
+ OutOfBandManagementOwnerSearch.and("server", OutOfBandManagementOwnerSearch.entity().getManagementServerId(), SearchCriteria.Op.EQ);
+ OutOfBandManagementOwnerSearch.or("serverNull", OutOfBandManagementOwnerSearch.entity().getManagementServerId(), SearchCriteria.Op.NULL);
+ OutOfBandManagementOwnerSearch.done();
+
+ StateUpdateSearch = createSearchBuilder();
+ StateUpdateSearch.and("status", StateUpdateSearch.entity().getPowerState(), SearchCriteria.Op.EQ);
+ StateUpdateSearch.and("id", StateUpdateSearch.entity().getId(), SearchCriteria.Op.EQ);
+ StateUpdateSearch.and("update", StateUpdateSearch.entity().getUpdateCount(), SearchCriteria.Op.EQ);
+ StateUpdateSearch.done();
+
+ PowerStateAttr = _allAttributes.get("powerState");
+ MsIdAttr = _allAttributes.get("managementServerId");
+ UpdateTimeAttr = _allAttributes.get("updateTime");
+ assert (PowerStateAttr != null && MsIdAttr != null && UpdateTimeAttr != null) : "Couldn't find one of these attributes";
+ }
+
+ @Override
+ public OutOfBandManagement findByHost(long hostId) {
+ SearchCriteria<OutOfBandManagementVO> sc = HostSearch.create("hostId", hostId);
+ return findOneBy(sc);
+ }
+
+ @Override
+ public List<OutOfBandManagementVO> findAllByManagementServer(long serverId) {
+ SearchCriteria<OutOfBandManagementVO> sc = OutOfBandManagementOwnerSearch.create();
+ sc.setParameters("server", serverId);
+ return listBy(sc, new Filter(OutOfBandManagementVO.class, "updateTime", true, null, null));
+ }
+
+ private void executeExpireOwnershipSql(final String sql, long resource) {
+ TransactionLegacy txn = TransactionLegacy.currentTxn();
+ try {
--- End diff --
Fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by nvazquez <gi...@git.apache.org>.
Github user nvazquez commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-218468571
@rhtyd sure, I'll examine this. This tests are introduced in #1497, I'll work on this
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r62444406
--- Diff: utils/src/main/java/org/apache/cloudstack/utils/process/ProcessRunner.java ---
@@ -0,0 +1,111 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+
+package org.apache.cloudstack.utils.process;
+
+import org.apache.log4j.Logger;
+
+import java.io.IOException;
+import java.util.List;
+
+public class ProcessRunner {
+ public static final Logger LOG = Logger.getLogger(ProcessRunner.class);
+
+ private String stdOutput;
+ private String stdError;
+ private int returnCode = -1;
+
+ public String getStdOutput() {
+ return stdOutput;
+ }
+
+ public void setStdOutput(String stdOutput) {
+ this.stdOutput = stdOutput;
+ }
+
+ public String getStdError() {
+ return stdError;
+ }
+
+ public void setStdError(String stdError) {
+ this.stdError = stdError;
+ }
+
+ public int getReturnCode() {
+ return returnCode;
+ }
+
+ public void setReturnCode(int returnCode) {
+ this.returnCode = returnCode;
+ }
+
+ public static ProcessRunner executeCommands(List<String> commands, long timeOutSeconds) {
+ ProcessRunner result = new ProcessRunner();
+ try {
+ Process process = new ProcessBuilder().command(commands).start();
+ StreamGobbler stdInputGobbler = new StreamGobbler(process.getInputStream());
+ StreamGobbler stdErrorGobbler = new StreamGobbler(process.getErrorStream());
--- End diff --
inheritIO is not useful here as we want to capture the input/output stream in stdErr and stdOut. Using future and a cache thread pool executor, I've removed the explicit use of a stream gobbler here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by serg38 <gi...@git.apache.org>.
Github user serg38 commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-218543261
@swill PR1539 passed Jenkins and Travis. After you merge it should resolve the issue in other PRs
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61815144
--- Diff: test/integration/smoke/test_outofbandmanagement.py ---
@@ -0,0 +1,561 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import marvin
+from marvin.cloudstackTestCase import *
+from marvin.cloudstackAPI import *
+from marvin.lib.utils import *
+from marvin.lib.base import *
+from marvin.lib.common import *
+from marvin.lib.utils import (random_gen)
+from nose.plugins.attrib import attr
+
+from ipmisim.ipmisim import IpmiServerContext, IpmiServer, ThreadedIpmiServer
+
+import socket
+import sys
+import thread
+import time
+
+
+class TestOutOfBandManagement(cloudstackTestCase):
+ """ Test cases for out of band management
+ """
+
+ def setUp(self):
+ self.apiclient = self.testClient.getApiClient()
+ self.hypervisor = self.testClient.getHypervisorInfo()
+ self.dbclient = self.testClient.getDbConnection()
+ self.services = self.testClient.getParsedTestDataConfig()
+ self.mgtSvrDetails = self.config.__dict__["mgtSvr"][0].__dict__
+
+ self.zone = get_zone(self.apiclient, self.testClient.getZoneForTests())
+ self.host = None
+ self.server = None
+
+ # use random port for ipmisim
+ s = socket.socket()
+ s.bind(('', 0))
+ self.serverPort = s.getsockname()[1]
+ s.close()
+
+ self.cleanup = []
+
+
+ def tearDown(self):
+ try:
+ self.dbclient.execute("delete from oobm where port=%d" % self.getIpmiServerPort())
+ self.dbclient.execute("delete from mshost_peer where peer_runid=%s" % self.getFakeMsRunId())
+ self.dbclient.execute("delete from mshost where runid=%s" % self.getFakeMsRunId())
+ self.dbclient.execute("delete from cluster_details where name='outOfBandManagementEnabled'")
+ self.dbclient.execute("delete from data_center_details where name='outOfBandManagementEnabled'")
+ cleanup_resources(self.apiclient, self.cleanup)
+ if self.server:
+ self.server.shutdown()
+ self.server.server_close()
+ except Exception as e:
+ raise Exception("Warning: Exception during cleanup : %s" % e)
+
+
+ def getFakeMsId(self):
+ return 1234567890
--- End diff --
Consider using a random value to allow multiple executions of this test case to occur without impacting each other in the event that the cleanup does not complete successfully.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by DaanHoogland <gi...@git.apache.org>.
Github user DaanHoogland commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r60726125
--- Diff: api/src/org/apache/cloudstack/api/response/ZoneResponse.java ---
@@ -233,6 +234,9 @@ public void addTag(ResourceTagResponse tag) {
}
public void setResourceDetails(Map<String, String> details) {
- this.resourceDetails = details;
+ if (details == null) {
--- End diff --
are you solving an extra bug here? seems like it is no longer possible to set details to null. I suppose it is intended.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-213618402
@DaanHoogland additional notes on how tests work; we've introduce a new dependency `ipmisim` that I wrote for this feature, ipmisim is both a tool and a library that simulates an ipmi 2.0 service; we use ipmisim to spin up fake ipmi bmcs and configure them to a host. The marvin test uses that so we can run the tests without explicitly requiring an actual iLo/iDRAC bmc. `ipmisim`: https://pypi.python.org/pypi/ipmisim/0.6
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61798443
--- Diff: server/src/org/apache/cloudstack/outofbandmanagement/OutOfBandManagementServiceImpl.java ---
@@ -0,0 +1,532 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.outofbandmanagement;
+
+import com.cloud.alert.AlertManager;
+import com.cloud.dc.ClusterDetailsDao;
+import com.cloud.dc.ClusterDetailsVO;
+import com.cloud.dc.DataCenter;
+import com.cloud.dc.DataCenterDetailVO;
+import com.cloud.dc.dao.DataCenterDetailsDao;
+import com.cloud.domain.Domain;
+import com.cloud.event.ActionEvent;
+import com.cloud.event.ActionEventUtils;
+import com.cloud.event.EventTypes;
+import com.cloud.host.Host;
+import com.cloud.host.dao.HostDao;
+import com.cloud.org.Cluster;
+import com.cloud.utils.component.Manager;
+import com.cloud.utils.component.ManagerBase;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.db.Transaction;
+import com.cloud.utils.db.TransactionCallback;
+import com.cloud.utils.db.TransactionStatus;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.fsm.NoTransitionException;
+import com.google.common.base.Strings;
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import com.google.common.collect.ImmutableMap;
+import org.apache.cloudstack.api.response.OutOfBandManagementResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.framework.config.ConfigKey;
+import org.apache.cloudstack.framework.config.Configurable;
+import org.apache.cloudstack.outofbandmanagement.dao.OutOfBandManagementDao;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverChangePasswordCommand;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverPowerCommand;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverResponse;
+import org.apache.cloudstack.utils.identity.ManagementServerNode;
+import org.apache.log4j.Logger;
+import org.springframework.stereotype.Component;
+
+import javax.ejb.Local;
+import javax.inject.Inject;
+import javax.naming.ConfigurationException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+@Component
+@Local(value = {OutOfBandManagementService.class})
+public class OutOfBandManagementServiceImpl extends ManagerBase implements OutOfBandManagementService, Manager, Configurable {
+ public static final Logger LOG = Logger.getLogger(OutOfBandManagementServiceImpl.class);
+
+ @Inject
+ private ClusterDetailsDao clusterDetailsDao;
+ @Inject
+ private DataCenterDetailsDao dataCenterDetailsDao;
+ @Inject
+ private OutOfBandManagementDao outOfBandManagementDao;
+ @Inject
+ private HostDao hostDao;
+ @Inject
+ private AlertManager alertMgr;
+
+ private String name;
+ private long serviceId;
+
+ private List<OutOfBandManagementDriver> outOfBandManagementDrivers = new ArrayList<>();
+ private Map<String, OutOfBandManagementDriver> outOfBandManagementDriversMap = new HashMap<String, OutOfBandManagementDriver>();
+
+ private static final String OOBM_ENABLED_DETAIL = "outOfBandManagementEnabled";
+ private static final int ACQUIRE_GLOBAL_LOCK_TIMEOUT_FOR_HOST = 120;
+
+ private Cache<Long, Long> hostAlertCache;
+ private static ExecutorService backgroundSyncExecutor;
+
+ private String getOutOfBandManagementHostLock(long id) {
+ return "oobm.host." + id;
+ }
+
+ private void initializeDriversMap() {
+ if (outOfBandManagementDriversMap.isEmpty() && outOfBandManagementDrivers != null && outOfBandManagementDrivers.size() > 0) {
+ for (final OutOfBandManagementDriver driver : outOfBandManagementDrivers) {
+ outOfBandManagementDriversMap.put(driver.getName().toLowerCase(), driver);
+ }
+ LOG.debug("Discovered out-of-band management drivers configured in the OutOfBandManagementService");
+ }
+ }
+
+ private OutOfBandManagementDriver getDriver(final OutOfBandManagement outOfBandManagementConfig) {
+ if (!Strings.isNullOrEmpty(outOfBandManagementConfig.getDriver())) {
+ final OutOfBandManagementDriver driver = outOfBandManagementDriversMap.get(outOfBandManagementConfig.getDriver());
+ if (driver != null) {
+ return driver;
+ }
+ }
+ throw new CloudRuntimeException("Configured out-of-band management driver is not available. Aborting any out-of-band management action.");
+ }
+
+ protected OutOfBandManagement updateConfig(final OutOfBandManagement outOfBandManagementConfig, final ImmutableMap<OutOfBandManagement.Option, String> options) {
+ if (outOfBandManagementConfig == null) {
+ throw new CloudRuntimeException("Out-of-band management is not configured for the host. Aborting.");
+ }
+ if (options == null) {
+ return outOfBandManagementConfig;
+ }
+ for (OutOfBandManagement.Option option: options.keySet()) {
+ final String value = options.get(option);
+ if (Strings.isNullOrEmpty(value)) {
+ continue;
+ }
+ switch (option) {
+ case DRIVER:
+ outOfBandManagementConfig.setDriver(value);
+ break;
+ case ADDRESS:
+ outOfBandManagementConfig.setAddress(value);
+ break;
+ case PORT:
+ outOfBandManagementConfig.setPort(Integer.parseInt(value));
+ break;
+ case USERNAME:
+ outOfBandManagementConfig.setUsername(value);
+ break;
+ case PASSWORD:
+ outOfBandManagementConfig.setPassword(value);
+ break;
+ }
+ }
+ return outOfBandManagementConfig;
+ }
+
+ protected ImmutableMap<OutOfBandManagement.Option, String> getOptions(final OutOfBandManagement outOfBandManagementConfig) {
+ final ImmutableMap.Builder<OutOfBandManagement.Option, String> optionsBuilder = ImmutableMap.builder();
+ if (outOfBandManagementConfig == null) {
+ throw new CloudRuntimeException("Out-of-band management is not configured for the host. Aborting.");
+ }
+ for (OutOfBandManagement.Option option: OutOfBandManagement.Option.values()) {
+ String value = null;
+ switch (option) {
+ case DRIVER:
+ value = outOfBandManagementConfig.getDriver();
+ break;
+ case ADDRESS:
+ value = outOfBandManagementConfig.getAddress();
+ break;
+ case PORT:
+ if (outOfBandManagementConfig.getPort() != null) {
+ value = String.valueOf(outOfBandManagementConfig.getPort());
+ }
+ break;
+ case USERNAME:
+ value = outOfBandManagementConfig.getUsername();
+ break;
+ case PASSWORD:
+ value = outOfBandManagementConfig.getPassword();
+ break;
+ }
+ if (value != null) {
+ optionsBuilder.put(option, value);
+ }
+ }
+ return optionsBuilder.build();
+ }
+
+ private void sendAuthError(final Host host, final String message) {
+ try {
+ hostAlertCache.asMap().putIfAbsent(host.getId(), 0L);
+ Long sentCount = hostAlertCache.asMap().get(host.getId());
+ if (sentCount != null && sentCount <= 0) {
+ boolean concurrentUpdateResult = hostAlertCache.asMap().replace(host.getId(), sentCount, sentCount+1L);
+ if (concurrentUpdateResult) {
+ final String subject = String.format("Out-of-band management auth-error detected for host:%d in cluster:%d, zone:%d", host.getId(), host.getClusterId(), host.getDataCenterId());
+ LOG.error(subject + ": " + message);
+ alertMgr.sendAlert(AlertManager.AlertType.ALERT_TYPE_OOBM_AUTH_ERROR, host.getDataCenterId(), host.getPodId(), subject, message);
+ }
+ }
+ } catch (Exception ignored) {
+ }
+ }
+
+ private boolean transitionPowerState(OutOfBandManagement.PowerState.Event event, OutOfBandManagement outOfBandManagementHost) {
+ if (outOfBandManagementHost == null) {
+ return false;
+ }
+ OutOfBandManagement.PowerState currentPowerState = outOfBandManagementHost.getPowerState();
+ try {
+ OutOfBandManagement.PowerState newPowerState = OutOfBandManagement.PowerState.getStateMachine().getNextState(currentPowerState, event);
+ boolean result = outOfBandManagementDao.updateState(currentPowerState, event, newPowerState, outOfBandManagementHost, null);
+ if (result) {
+ final String message = String.format("Transitioned out-of-band management power state from:%s to:%s due to event:%s for the host id:%d", currentPowerState, newPowerState, event, outOfBandManagementHost.getHostId());
+ LOG.debug(message);
+ ActionEventUtils.onActionEvent(CallContext.current().getCallingUserId(), CallContext.current().getCallingAccountId(), Domain.ROOT_DOMAIN,
+ EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_POWERSTATE_TRANSITION, message);
+ }
+ return result;
+ } catch (NoTransitionException ignored) {
+ LOG.trace(String.format("Unable to transition out-of-band management power state for host id=%s for the event=%s and current power state=%s", outOfBandManagementHost.getHostId(), event, currentPowerState));
--- End diff --
Consider logging to ``ERROR`` as the system is not behaving as specified and in a completely unexpected manner. Also, please include the stack trace in the log message to assist with debugging.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61846419
--- Diff: server/src/org/apache/cloudstack/outofbandmanagement/OutOfBandManagementServiceImpl.java ---
@@ -0,0 +1,532 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.outofbandmanagement;
+
+import com.cloud.alert.AlertManager;
+import com.cloud.dc.ClusterDetailsDao;
+import com.cloud.dc.ClusterDetailsVO;
+import com.cloud.dc.DataCenter;
+import com.cloud.dc.DataCenterDetailVO;
+import com.cloud.dc.dao.DataCenterDetailsDao;
+import com.cloud.domain.Domain;
+import com.cloud.event.ActionEvent;
+import com.cloud.event.ActionEventUtils;
+import com.cloud.event.EventTypes;
+import com.cloud.host.Host;
+import com.cloud.host.dao.HostDao;
+import com.cloud.org.Cluster;
+import com.cloud.utils.component.Manager;
+import com.cloud.utils.component.ManagerBase;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.db.Transaction;
+import com.cloud.utils.db.TransactionCallback;
+import com.cloud.utils.db.TransactionStatus;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.fsm.NoTransitionException;
+import com.google.common.base.Strings;
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import com.google.common.collect.ImmutableMap;
+import org.apache.cloudstack.api.response.OutOfBandManagementResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.framework.config.ConfigKey;
+import org.apache.cloudstack.framework.config.Configurable;
+import org.apache.cloudstack.outofbandmanagement.dao.OutOfBandManagementDao;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverChangePasswordCommand;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverPowerCommand;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverResponse;
+import org.apache.cloudstack.utils.identity.ManagementServerNode;
+import org.apache.log4j.Logger;
+import org.springframework.stereotype.Component;
+
+import javax.ejb.Local;
+import javax.inject.Inject;
+import javax.naming.ConfigurationException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+@Component
+@Local(value = {OutOfBandManagementService.class})
--- End diff --
removed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r62569193
--- Diff: utils/src/main/java/org/apache/cloudstack/utils/process/ProcessRunner.java ---
@@ -0,0 +1,112 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+
+package org.apache.cloudstack.utils.process;
+
+import com.cloud.utils.concurrency.NamedThreadFactory;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Strings;
+import org.apache.log4j.Logger;
+import org.joda.time.Duration;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.List;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+public class ProcessRunner {
+ public static final Logger LOG = Logger.getLogger(ProcessRunner.class);
+
+ private static final ExecutorService processExecutor = Executors.newCachedThreadPool(new NamedThreadFactory("ProcessRunner"));
+
+ private static String readStream(final InputStream inputStream) {
+ String text = null;
+ try {
+ final BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(inputStream));
+ String line;
--- End diff --
Convert to a ``StringBuilder`` to avoid unnecessary String re-allocation.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r62569864
--- Diff: utils/src/main/java/org/apache/cloudstack/utils/process/ProcessRunner.java ---
@@ -0,0 +1,112 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+
+package org.apache.cloudstack.utils.process;
+
+import com.cloud.utils.concurrency.NamedThreadFactory;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Strings;
+import org.apache.log4j.Logger;
+import org.joda.time.Duration;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.List;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+public class ProcessRunner {
+ public static final Logger LOG = Logger.getLogger(ProcessRunner.class);
+
+ private static final ExecutorService processExecutor = Executors.newCachedThreadPool(new NamedThreadFactory("ProcessRunner"));
+
+ private static String readStream(final InputStream inputStream) {
+ String text = null;
+ try {
+ final BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(inputStream));
+ String line;
+ while ((line = bufferedReader.readLine()) != null) {
+ if (Strings.isNullOrEmpty(text)) {
+ text = line;
+ } else {
+ text = text + "\n" + line;
+ }
+ }
+ } catch (IOException e) {
+ if (LOG.isTraceEnabled()) {
+ LOG.trace("ProcessRunner::readStream failed due to: " + e.getMessage());
+ }
+ }
+ return text;
+ }
+
+ public static ProcessResult executeCommands(final List<String> commands, final Duration timeOut) {
+ Preconditions.checkArgument(commands != null && timeOut != null);
+
+ int retVal = -2;
+ String stdOutput = null;
+ String stdError = null;
+
+ try {
+ final Process process = new ProcessBuilder().command(commands).start();
+ if (timeOut.getStandardSeconds() > 0) {
+ final Future<Integer> processFuture = processExecutor.submit(new Callable<Integer>() {
+ @Override
+ public Integer call() throws Exception {
+ return process.waitFor();
+ }
+ });
+ try {
+ retVal = processFuture.get(timeOut.getStandardSeconds(), TimeUnit.SECONDS);
+ } catch (ExecutionException | TimeoutException e) {
--- End diff --
Why are signaling a failure with a return value instead of an exception?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r62322683
--- Diff: utils/src/main/java/org/apache/cloudstack/utils/process/ProcessRunner.java ---
@@ -0,0 +1,111 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+
+package org.apache.cloudstack.utils.process;
+
+import org.apache.log4j.Logger;
+
+import java.io.IOException;
+import java.util.List;
+
+public class ProcessRunner {
+ public static final Logger LOG = Logger.getLogger(ProcessRunner.class);
+
+ private String stdOutput;
+ private String stdError;
+ private int returnCode = -1;
+
+ public String getStdOutput() {
+ return stdOutput;
+ }
+
+ public void setStdOutput(String stdOutput) {
+ this.stdOutput = stdOutput;
+ }
+
+ public String getStdError() {
+ return stdError;
+ }
+
+ public void setStdError(String stdError) {
+ this.stdError = stdError;
+ }
+
+ public int getReturnCode() {
+ return returnCode;
+ }
+
+ public void setReturnCode(int returnCode) {
+ this.returnCode = returnCode;
+ }
+
+ public static ProcessRunner executeCommands(List<String> commands, long timeOutSeconds) {
--- End diff --
Consider adding ``Preconditions.checkArguments`` to verify that ``commands`` is not ``null``.
Also, consider using JodaTime's ``Duration`` to represent the timeout as number with a unit of measure. This step is not required for LGTM, but would be a good refactoring if we fit it in.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r62441858
--- Diff: server/src/com/cloud/server/StatsCollector.java ---
@@ -412,6 +428,36 @@ protected void runInContext() {
}
}
+ class HostOutOfBandManagementStatsCollector extends ManagedContextRunnable {
+ @Override
+ protected void runInContext() {
+ try {
+ s_logger.debug("HostOutOfBandManagementStatsCollector is running...");
+ List<OutOfBandManagementVO> outOfBandManagementHosts = outOfBandManagementDao.findAllByManagementServer(ManagementServerNode.getManagementServerId());
+ if (outOfBandManagementHosts == null) {
+ return;
+ }
+ for (OutOfBandManagement outOfBandManagementHost : outOfBandManagementHosts) {
+ Host host = _hostDao.findById(outOfBandManagementHost.getHostId());
+ if (host == null) {
+ continue;
+ }
+ if (outOfBandManagementService.isOutOfBandManagementEnabled(host)) {
+ outOfBandManagementService.submitBackgroundPowerSyncTask(host);
+ } else {
+ if (outOfBandManagementHost.getPowerState() != OutOfBandManagement.PowerState.Disabled) {
+ if (outOfBandManagementService.transitionPowerStateToDisabled(Collections.singletonList(host))) {
--- End diff --
Fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r62570445
--- Diff: utils/src/main/java/org/apache/cloudstack/utils/process/ProcessRunner.java ---
@@ -0,0 +1,112 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+
+package org.apache.cloudstack.utils.process;
+
+import com.cloud.utils.concurrency.NamedThreadFactory;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Strings;
+import org.apache.log4j.Logger;
+import org.joda.time.Duration;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.List;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+public class ProcessRunner {
+ public static final Logger LOG = Logger.getLogger(ProcessRunner.class);
+
+ private static final ExecutorService processExecutor = Executors.newCachedThreadPool(new NamedThreadFactory("ProcessRunner"));
--- End diff --
A cached thread pool is unbounded. Should we consider the use of a bounded thread pool to prevent shell processes from overwhelming the management server?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r62442464
--- Diff: utils/src/main/java/org/apache/cloudstack/utils/process/ProcessRunner.java ---
@@ -0,0 +1,111 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+
+package org.apache.cloudstack.utils.process;
+
+import org.apache.log4j.Logger;
+
+import java.io.IOException;
+import java.util.List;
+
+public class ProcessRunner {
+ public static final Logger LOG = Logger.getLogger(ProcessRunner.class);
+
+ private String stdOutput;
+ private String stdError;
+ private int returnCode = -1;
--- End diff --
Fixed, made members final.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-217434570
@rhtyd @swill we are very close. I have a couple of outstanding comments -- particularly around potential thread explosion in ``ProcessRunner``. Ultimately, I think the issues are straightforward to address.
/cc @borisstoyanov
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-214233279
@wido thanks for the comments. It seemed easier to pass the arguments to the oobm subsystem, there is a immutable map of option,string being returned this is why all configuration arguments are string. Also, the driver would convert the port to a string to execute this in a forked process (that calls `ipmitool`). For this reason, all configs are passed as strings and left to how the driver would like to use them or interpret them.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-216309500
@swill I have gotten diverted from review. Hoping to pick up today.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61777563
--- Diff: api/src/org/apache/cloudstack/api/command/admin/outofbandmanagement/IssueOutOfBandManagementPowerActionCmd.java ---
@@ -0,0 +1,128 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.api.command.admin.outofbandmanagement;
+
+import com.cloud.event.EventTypes;
+import com.cloud.exception.ConcurrentOperationException;
+import com.cloud.exception.InsufficientCapacityException;
+import com.cloud.exception.NetworkRuleConflictException;
+import com.cloud.exception.ResourceAllocationException;
+import com.cloud.exception.ResourceUnavailableException;
+import com.cloud.host.Host;
+import com.google.common.base.Strings;
+import org.apache.cloudstack.acl.RoleType;
+import org.apache.cloudstack.api.APICommand;
+import org.apache.cloudstack.api.ApiCommandJobType;
+import org.apache.cloudstack.api.ApiConstants;
+import org.apache.cloudstack.api.ApiErrorCode;
+import org.apache.cloudstack.api.BaseAsyncCmd;
+import org.apache.cloudstack.api.Parameter;
+import org.apache.cloudstack.api.ServerApiException;
+import org.apache.cloudstack.api.response.HostResponse;
+import org.apache.cloudstack.api.response.OutOfBandManagementResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagement.PowerOperation;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagementService;
+
+import javax.inject.Inject;
+
+@APICommand(name = "issueOutOfBandManagementPowerAction", description = "Initiates the specified power action to the host's out-of-band management interface",
+ responseObject = OutOfBandManagementResponse.class, requestHasSensitiveInfo = false, responseHasSensitiveInfo = false, authorized = {RoleType.Admin})
+public class IssueOutOfBandManagementPowerActionCmd extends BaseAsyncCmd {
+ @Inject
+ private OutOfBandManagementService outOfBandManagementService;
+
+ /////////////////////////////////////////////////////
+ //////////////// API parameters /////////////////////
+ /////////////////////////////////////////////////////
+
+ @Parameter(name = ApiConstants.HOST_ID, type = CommandType.UUID, entityType = HostResponse.class, required = true, description = "the ID of the host")
+ private Long hostId;
+
+ @Parameter(name = ApiConstants.TIMEOUT, type = CommandType.LONG, description = "optional operation timeout in seconds that overrides the global or cluster-level out-of-band management timeout setting")
+ private Long actionTimeout;
+
+ @Parameter(name = ApiConstants.ACTION, type = CommandType.STRING, required = true, description = "out-of-band management power actions, valid actions are: ON, OFF, CYCLE, RESET, SOFT, STATUS")
+ private String outOfBandManagementPowerOperation;
+
+ /////////////////////////////////////////////////////
+ /////////////// API Implementation///////////////////
+ /////////////////////////////////////////////////////
+
+ @Override
+ public String getCommandName() {
+ return "issueoutofbandmanagementpoweractionresponse";
+ }
+
+ private void validateParams() {
+ if (getHostId() == null || getHostId() < 1L) {
+ throw new ServerApiException(ApiErrorCode.PARAM_ERROR, "Invalid host ID: " + getHostId());
+ }
+ if (Strings.isNullOrEmpty(getOutOfBandManagementPowerOperation())) {
+ throw new ServerApiException(ApiErrorCode.PARAM_ERROR, "Invalid out-of-band management power action: " + getOutOfBandManagementPowerOperation());
+ }
--- End diff --
Add a check that the ``outOfBandManagementPowerOperation`` value a member of the ``PowerOperation`` enumeration.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r62321985
--- Diff: server/src/com/cloud/server/StatsCollector.java ---
@@ -412,6 +428,36 @@ protected void runInContext() {
}
}
+ class HostOutOfBandManagementStatsCollector extends ManagedContextRunnable {
+ @Override
+ protected void runInContext() {
+ try {
+ s_logger.debug("HostOutOfBandManagementStatsCollector is running...");
+ List<OutOfBandManagementVO> outOfBandManagementHosts = outOfBandManagementDao.findAllByManagementServer(ManagementServerNode.getManagementServerId());
+ if (outOfBandManagementHosts == null) {
+ return;
+ }
+ for (OutOfBandManagement outOfBandManagementHost : outOfBandManagementHosts) {
+ Host host = _hostDao.findById(outOfBandManagementHost.getHostId());
+ if (host == null) {
+ continue;
+ }
+ if (outOfBandManagementService.isOutOfBandManagementEnabled(host)) {
+ outOfBandManagementService.submitBackgroundPowerSyncTask(host);
+ } else {
+ if (outOfBandManagementHost.getPowerState() != OutOfBandManagement.PowerState.Disabled) {
+ if (outOfBandManagementService.transitionPowerStateToDisabled(Collections.singletonList(host))) {
+ s_logger.debug("Out-of-band management was disabled in zone/cluster/host, disabled power state for host id:" + host.getId());
--- End diff --
Please wrap in a ``if (LOGGER.isDebugEnabled)`` check.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r62655962
--- Diff: utils/src/main/java/org/apache/cloudstack/utils/process/ProcessRunner.java ---
@@ -0,0 +1,109 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+
+package org.apache.cloudstack.utils.process;
+
+import com.cloud.utils.concurrency.NamedThreadFactory;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Strings;
+import org.apache.log4j.Logger;
+import org.joda.time.Duration;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.List;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+public class ProcessRunner {
+ public static final Logger LOG = Logger.getLogger(ProcessRunner.class);
+
+ private static final ExecutorService processExecutor = Executors.newCachedThreadPool(new NamedThreadFactory("ProcessRunner"));
+
+ private static String readStream(final InputStream inputStream) throws IOException {
+ final StringBuilder sb = new StringBuilder();
+ final BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(inputStream));
+ String line;
+ while ((line = bufferedReader.readLine()) != null) {
+ sb.append(line);
+ sb.append("\n");
+ }
+ return sb.toString();
+ }
+
+ public static ProcessResult executeCommands(final List<String> commands, final Duration timeOut) {
+ Preconditions.checkArgument(commands != null && timeOut != null);
+
+ int retVal = -2;
+ String stdOutput = null;
+ String stdError = null;
+
+ try {
+ final Process process = new ProcessBuilder().command(commands).start();
+ if (timeOut.getStandardSeconds() > 0) {
+ final Future<Integer> processFuture = processExecutor.submit(new Callable<Integer>() {
+ @Override
+ public Integer call() throws Exception {
+ return process.waitFor();
+ }
+ });
+ try {
+ retVal = processFuture.get(timeOut.getStandardSeconds(), TimeUnit.SECONDS);
+ } catch (ExecutionException e) {
+ retVal = -1;
+ stdError = e.getMessage();
+ if (LOG.isTraceEnabled()) {
+ LOG.trace("Failed to complete the requested command due to execution error: " + e.getMessage());
+ }
+ } catch (TimeoutException e) {
+ retVal = -1;
+ stdError = "Operation timed out, aborted";
+ if (LOG.isTraceEnabled()) {
+ LOG.trace("Failed to complete the requested command within timeout: " + e.getMessage());
+ }
+ } finally {
+ if (Strings.isNullOrEmpty(stdError)) {
+ stdOutput = readStream(process.getInputStream());
+ stdError = readStream(process.getErrorStream());
--- End diff --
Have you considered using Guava's ``CharStreams.toString(new BufferedWriter(process.getErrorStream))`` instead of ``readStream``?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: [WIP] Don't start review yet -- CLOUDSTAC...
Posted by bhaisaab <gi...@git.apache.org>.
Github user bhaisaab commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-212338195
Local test results (the feature is hypervisor agnostic, integration test also run with Travis as well):
==== Marvin Init Successful ====
=== TestName: test_01_configure_oobm_invalid | Status : SUCCESS ===
=== TestName: test_02_configure_oobm_valid | Status : SUCCESS ===
=== TestName: test_03_enabledisable_oobm_invalid | Status : SUCCESS ===
=== TestName: test_04_enabledisable_oobm_valid | Status : SUCCESS ===
=== TestName: test_05_enabledisable_across_clusterzones_oobm_valid | Status : SUCCESS ===
=== TestName: test_06_oobm_issue_power_action | Status : SUCCESS ===
=== TestName: test_07_oobm_background_powerstate_sync | Status : SUCCESS ===
=== TestName: test_08_multiple_mgmt_server_ownership | Status : SUCCESS ===
=== TestName: test_09_oobm_change_password | Status : SUCCESS ===
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by swill <gi...@git.apache.org>.
Github user swill commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-214797513
I think this PR is in pretty good order. I will let @jburwell have a look at it to see if he has any comments because it is a large PR, but pending any of his feedback, I think we are set on this one...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by bhaisaab <gi...@git.apache.org>.
Github user bhaisaab commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-213395335
@DaanHoogland thanks, @borisstoyanov will share QA results next week as well
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61850255
--- Diff: test/integration/smoke/test_outofbandmanagement.py ---
@@ -0,0 +1,561 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import marvin
+from marvin.cloudstackTestCase import *
+from marvin.cloudstackAPI import *
+from marvin.lib.utils import *
+from marvin.lib.base import *
+from marvin.lib.common import *
+from marvin.lib.utils import (random_gen)
+from nose.plugins.attrib import attr
+
+from ipmisim.ipmisim import IpmiServerContext, IpmiServer, ThreadedIpmiServer
+
+import socket
+import sys
+import thread
+import time
+
+
+class TestOutOfBandManagement(cloudstackTestCase):
+ """ Test cases for out of band management
+ """
+
+ def setUp(self):
+ self.apiclient = self.testClient.getApiClient()
+ self.hypervisor = self.testClient.getHypervisorInfo()
+ self.dbclient = self.testClient.getDbConnection()
+ self.services = self.testClient.getParsedTestDataConfig()
+ self.mgtSvrDetails = self.config.__dict__["mgtSvr"][0].__dict__
+
+ self.zone = get_zone(self.apiclient, self.testClient.getZoneForTests())
+ self.host = None
+ self.server = None
+
+ # use random port for ipmisim
+ s = socket.socket()
+ s.bind(('', 0))
+ self.serverPort = s.getsockname()[1]
+ s.close()
+
+ self.cleanup = []
+
+
+ def tearDown(self):
+ try:
+ self.dbclient.execute("delete from oobm where port=%d" % self.getIpmiServerPort())
+ self.dbclient.execute("delete from mshost_peer where peer_runid=%s" % self.getFakeMsRunId())
+ self.dbclient.execute("delete from mshost where runid=%s" % self.getFakeMsRunId())
+ self.dbclient.execute("delete from cluster_details where name='outOfBandManagementEnabled'")
+ self.dbclient.execute("delete from data_center_details where name='outOfBandManagementEnabled'")
+ cleanup_resources(self.apiclient, self.cleanup)
+ if self.server:
+ self.server.shutdown()
+ self.server.server_close()
+ except Exception as e:
+ raise Exception("Warning: Exception during cleanup : %s" % e)
+
+
+ def getFakeMsId(self):
+ return 1234567890
--- End diff --
fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by kiwiflyer <gi...@git.apache.org>.
Github user kiwiflyer commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-213423589
Really exciting PR @bhaisaab!
We'll pull this in for testing as well.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-216472391
@jburwell I've fixed based on your review, please re-review and advise further improvements, thanks
Integration test after last PR review changes:
=== TestName: test_oobm_background_powerstate_sync | Status : SUCCESS ===
=== TestName: test_oobm_change_password | Status : SUCCESS ===
=== TestName: test_oobm_configure_default_driver | Status : SUCCESS ===
=== TestName: test_oobm_configure_invalid_driver | Status : SUCCESS ===
=== TestName: test_oobm_disable_feature_invalid | Status : SUCCESS ===
=== TestName: test_oobm_disable_feature_valid | Status : SUCCESS ===
=== TestName: test_oobm_enable_feature_invalid | Status : SUCCESS ===
=== TestName: test_oobm_enable_feature_valid | Status : SUCCESS ===
=== TestName: test_oobm_enabledisable_across_clusterzones | Status : SUCCESS ===
=== TestName: test_oobm_issue_power_cycle | Status : SUCCESS ===
=== TestName: test_oobm_issue_power_off | Status : SUCCESS ===
=== TestName: test_oobm_issue_power_on | Status : SUCCESS ===
=== TestName: test_oobm_issue_power_reset | Status : SUCCESS ===
=== TestName: test_oobm_issue_power_soft | Status : SUCCESS ===
=== TestName: test_oobm_issue_power_status | Status : SUCCESS ===
=== TestName: test_oobm_multiple_mgmt_server_ownership | Status : SUCCESS ===
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by swill <gi...@git.apache.org>.
Github user swill commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-216274944
@jburwell I will be doing a group of merges in a couple of hours which will include this one. If you have any pending concerns with this one, speak up soon. :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r60890271
--- Diff: api/src/org/apache/cloudstack/api/command/admin/outofbandmanagement/ConfigureOutOfBandManagementCmd.java ---
@@ -0,0 +1,127 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.api.command.admin.outofbandmanagement;
+
+import com.cloud.exception.ConcurrentOperationException;
+import com.cloud.exception.InsufficientCapacityException;
+import com.cloud.exception.NetworkRuleConflictException;
+import com.cloud.exception.ResourceAllocationException;
+import com.cloud.exception.ResourceUnavailableException;
+import com.cloud.host.Host;
+import com.google.common.base.Strings;
+import com.google.common.collect.ImmutableMap;
+import org.apache.cloudstack.acl.RoleType;
+import org.apache.cloudstack.api.APICommand;
+import org.apache.cloudstack.api.ApiConstants;
+import org.apache.cloudstack.api.ApiErrorCode;
+import org.apache.cloudstack.api.BaseCmd;
+import org.apache.cloudstack.api.Parameter;
+import org.apache.cloudstack.api.ServerApiException;
+import org.apache.cloudstack.api.response.HostResponse;
+import org.apache.cloudstack.api.response.OutOfBandManagementResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagement;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagementService;
+
+import javax.inject.Inject;
+
+@APICommand(name = "configureOutOfBandManagement", description = "Configures a host's out-of-band management interface",
+ responseObject = OutOfBandManagementResponse.class, requestHasSensitiveInfo = true, responseHasSensitiveInfo = false, authorized = {RoleType.Admin})
+public class ConfigureOutOfBandManagementCmd extends BaseCmd {
+ @Inject
+ private OutOfBandManagementService outOfBandManagementService;
+
+ /////////////////////////////////////////////////////
+ //////////////// API parameters /////////////////////
+ /////////////////////////////////////////////////////
+
+ @Parameter(name = ApiConstants.HOST_ID, type = CommandType.UUID, entityType = HostResponse.class, required = true, description = "the ID of the host")
+ private Long hostId;
+
+ @Parameter(name = ApiConstants.DRIVER, type = CommandType.STRING, required = true, description = "the host management interface driver, for example: ipmitool")
+ private String driver;
+
+ @Parameter(name = ApiConstants.IP_ADDRESS, type = CommandType.STRING, required = true, description = "the host management interface IP address")
+ private String address;
+
+ @Parameter(name = ApiConstants.PORT, type = CommandType.STRING, required = true, description = "the host management interface port")
--- End diff --
For reasons described below, for now I'm keeping it a string.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by DaanHoogland <gi...@git.apache.org>.
Github user DaanHoogland commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-213616553
ok, @swill reporting without upr. two manual retests did make it equal in effort.
all regular bubble tests passed on this except for the two I never get right the first time.
[1502.results.network.txt](https://github.com/apache/cloudstack/files/232648/1502.results.network.txt)
[1502.results.vpc_router.txt](https://github.com/apache/cloudstack/files/232650/1502.results.vpc_router.txt)
retests:
[1502.results.routers_network_ops.txt](https://github.com/apache/cloudstack/files/232649/1502.results.routers_network_ops.txt)
[dahn@blimbing templates]$ ssh root@192.168.23.7
Warning: Permanently added '192.168.23.7' (ECDSA) to the list of known hosts.
root@192.168.23.7's password:
# ping -c 3 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=46 time=24.063 ms
64 bytes from 8.8.8.8: seq=1 ttl=46 time=65.177 ms
64 bytes from 8.8.8.8: seq=2 ttl=46 time=23.749 ms
--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 23.749/37.663/65.177 ms
I also started the tests added in this PR, though ILO technology is not really relevant in a virtualised environment. Some but not all of them passed, I don't think it should keep us from merging this experimental feature in. I will look at the configuration changes in advanced.cfg to see if I can get them to run in the 2kvm bubble cfg. probably not all of them.
[1502.results.outofbandmanagement.txt](https://github.com/apache/cloudstack/files/232653/1502.results.outofbandmanagement.txt)
@rhtyd maybe in a follow-up; can we split the test in hardware required true and false?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61843025
--- Diff: api/src/org/apache/cloudstack/api/command/admin/outofbandmanagement/EnableOutOfBandManagementForClusterCmd.java ---
@@ -0,0 +1,105 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.api.command.admin.outofbandmanagement;
+
+import com.cloud.event.EventTypes;
+import com.cloud.exception.ConcurrentOperationException;
+import com.cloud.exception.InsufficientCapacityException;
+import com.cloud.exception.NetworkRuleConflictException;
+import com.cloud.exception.ResourceAllocationException;
+import com.cloud.exception.ResourceUnavailableException;
+import com.cloud.org.Cluster;
+import org.apache.cloudstack.acl.RoleType;
+import org.apache.cloudstack.api.APICommand;
+import org.apache.cloudstack.api.ApiConstants;
+import org.apache.cloudstack.api.ApiErrorCode;
+import org.apache.cloudstack.api.BaseAsyncCmd;
+import org.apache.cloudstack.api.BaseCmd;
+import org.apache.cloudstack.api.Parameter;
+import org.apache.cloudstack.api.ServerApiException;
+import org.apache.cloudstack.api.response.ClusterResponse;
+import org.apache.cloudstack.api.response.OutOfBandManagementResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagementService;
+
+import javax.inject.Inject;
+
+@APICommand(name = "enableOutOfBandManagementForCluster", description = "Enables out-of-band management for a cluster",
--- End diff --
fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61846062
--- Diff: server/src/com/cloud/server/StatsCollector.java ---
@@ -251,8 +262,9 @@ public boolean start() {
}
private void init(Map<String, String> configs) {
- _executor = Executors.newScheduledThreadPool(4, new NamedThreadFactory("StatsCollector"));
+ _executor = Executors.newScheduledThreadPool(6, new NamedThreadFactory("StatsCollector"));
--- End diff --
max. number of statscollector jobs configured in the init(); see code
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61847380
--- Diff: server/src/org/apache/cloudstack/outofbandmanagement/OutOfBandManagementServiceImpl.java ---
@@ -0,0 +1,532 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.outofbandmanagement;
+
+import com.cloud.alert.AlertManager;
+import com.cloud.dc.ClusterDetailsDao;
+import com.cloud.dc.ClusterDetailsVO;
+import com.cloud.dc.DataCenter;
+import com.cloud.dc.DataCenterDetailVO;
+import com.cloud.dc.dao.DataCenterDetailsDao;
+import com.cloud.domain.Domain;
+import com.cloud.event.ActionEvent;
+import com.cloud.event.ActionEventUtils;
+import com.cloud.event.EventTypes;
+import com.cloud.host.Host;
+import com.cloud.host.dao.HostDao;
+import com.cloud.org.Cluster;
+import com.cloud.utils.component.Manager;
+import com.cloud.utils.component.ManagerBase;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.db.Transaction;
+import com.cloud.utils.db.TransactionCallback;
+import com.cloud.utils.db.TransactionStatus;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.fsm.NoTransitionException;
+import com.google.common.base.Strings;
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import com.google.common.collect.ImmutableMap;
+import org.apache.cloudstack.api.response.OutOfBandManagementResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.framework.config.ConfigKey;
+import org.apache.cloudstack.framework.config.Configurable;
+import org.apache.cloudstack.outofbandmanagement.dao.OutOfBandManagementDao;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverChangePasswordCommand;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverPowerCommand;
+import org.apache.cloudstack.outofbandmanagement.driver.OutOfBandManagementDriverResponse;
+import org.apache.cloudstack.utils.identity.ManagementServerNode;
+import org.apache.log4j.Logger;
+import org.springframework.stereotype.Component;
+
+import javax.ejb.Local;
+import javax.inject.Inject;
+import javax.naming.ConfigurationException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+@Component
+@Local(value = {OutOfBandManagementService.class})
+public class OutOfBandManagementServiceImpl extends ManagerBase implements OutOfBandManagementService, Manager, Configurable {
+ public static final Logger LOG = Logger.getLogger(OutOfBandManagementServiceImpl.class);
+
+ @Inject
+ private ClusterDetailsDao clusterDetailsDao;
+ @Inject
+ private DataCenterDetailsDao dataCenterDetailsDao;
+ @Inject
+ private OutOfBandManagementDao outOfBandManagementDao;
+ @Inject
+ private HostDao hostDao;
+ @Inject
+ private AlertManager alertMgr;
+
+ private String name;
+ private long serviceId;
+
+ private List<OutOfBandManagementDriver> outOfBandManagementDrivers = new ArrayList<>();
+ private Map<String, OutOfBandManagementDriver> outOfBandManagementDriversMap = new HashMap<String, OutOfBandManagementDriver>();
+
+ private static final String OOBM_ENABLED_DETAIL = "outOfBandManagementEnabled";
+ private static final int ACQUIRE_GLOBAL_LOCK_TIMEOUT_FOR_HOST = 120;
+
+ private Cache<Long, Long> hostAlertCache;
+ private static ExecutorService backgroundSyncExecutor;
+
+ private String getOutOfBandManagementHostLock(long id) {
+ return "oobm.host." + id;
+ }
+
+ private void initializeDriversMap() {
+ if (outOfBandManagementDriversMap.isEmpty() && outOfBandManagementDrivers != null && outOfBandManagementDrivers.size() > 0) {
+ for (final OutOfBandManagementDriver driver : outOfBandManagementDrivers) {
+ outOfBandManagementDriversMap.put(driver.getName().toLowerCase(), driver);
+ }
+ LOG.debug("Discovered out-of-band management drivers configured in the OutOfBandManagementService");
+ }
+ }
+
+ private OutOfBandManagementDriver getDriver(final OutOfBandManagement outOfBandManagementConfig) {
+ if (!Strings.isNullOrEmpty(outOfBandManagementConfig.getDriver())) {
+ final OutOfBandManagementDriver driver = outOfBandManagementDriversMap.get(outOfBandManagementConfig.getDriver());
+ if (driver != null) {
+ return driver;
+ }
+ }
+ throw new CloudRuntimeException("Configured out-of-band management driver is not available. Aborting any out-of-band management action.");
+ }
+
+ protected OutOfBandManagement updateConfig(final OutOfBandManagement outOfBandManagementConfig, final ImmutableMap<OutOfBandManagement.Option, String> options) {
+ if (outOfBandManagementConfig == null) {
+ throw new CloudRuntimeException("Out-of-band management is not configured for the host. Aborting.");
+ }
+ if (options == null) {
+ return outOfBandManagementConfig;
+ }
+ for (OutOfBandManagement.Option option: options.keySet()) {
+ final String value = options.get(option);
+ if (Strings.isNullOrEmpty(value)) {
+ continue;
+ }
+ switch (option) {
+ case DRIVER:
+ outOfBandManagementConfig.setDriver(value);
+ break;
+ case ADDRESS:
+ outOfBandManagementConfig.setAddress(value);
+ break;
+ case PORT:
+ outOfBandManagementConfig.setPort(Integer.parseInt(value));
+ break;
+ case USERNAME:
+ outOfBandManagementConfig.setUsername(value);
+ break;
+ case PASSWORD:
+ outOfBandManagementConfig.setPassword(value);
+ break;
+ }
+ }
+ return outOfBandManagementConfig;
+ }
+
+ protected ImmutableMap<OutOfBandManagement.Option, String> getOptions(final OutOfBandManagement outOfBandManagementConfig) {
+ final ImmutableMap.Builder<OutOfBandManagement.Option, String> optionsBuilder = ImmutableMap.builder();
+ if (outOfBandManagementConfig == null) {
+ throw new CloudRuntimeException("Out-of-band management is not configured for the host. Aborting.");
+ }
+ for (OutOfBandManagement.Option option: OutOfBandManagement.Option.values()) {
+ String value = null;
+ switch (option) {
+ case DRIVER:
+ value = outOfBandManagementConfig.getDriver();
+ break;
+ case ADDRESS:
+ value = outOfBandManagementConfig.getAddress();
+ break;
+ case PORT:
+ if (outOfBandManagementConfig.getPort() != null) {
+ value = String.valueOf(outOfBandManagementConfig.getPort());
+ }
+ break;
+ case USERNAME:
+ value = outOfBandManagementConfig.getUsername();
+ break;
+ case PASSWORD:
+ value = outOfBandManagementConfig.getPassword();
+ break;
+ }
+ if (value != null) {
+ optionsBuilder.put(option, value);
+ }
+ }
+ return optionsBuilder.build();
+ }
+
+ private void sendAuthError(final Host host, final String message) {
+ try {
+ hostAlertCache.asMap().putIfAbsent(host.getId(), 0L);
+ Long sentCount = hostAlertCache.asMap().get(host.getId());
+ if (sentCount != null && sentCount <= 0) {
+ boolean concurrentUpdateResult = hostAlertCache.asMap().replace(host.getId(), sentCount, sentCount+1L);
+ if (concurrentUpdateResult) {
+ final String subject = String.format("Out-of-band management auth-error detected for host:%d in cluster:%d, zone:%d", host.getId(), host.getClusterId(), host.getDataCenterId());
+ LOG.error(subject + ": " + message);
+ alertMgr.sendAlert(AlertManager.AlertType.ALERT_TYPE_OOBM_AUTH_ERROR, host.getDataCenterId(), host.getPodId(), subject, message);
+ }
+ }
+ } catch (Exception ignored) {
+ }
+ }
+
+ private boolean transitionPowerState(OutOfBandManagement.PowerState.Event event, OutOfBandManagement outOfBandManagementHost) {
+ if (outOfBandManagementHost == null) {
+ return false;
+ }
+ OutOfBandManagement.PowerState currentPowerState = outOfBandManagementHost.getPowerState();
+ try {
+ OutOfBandManagement.PowerState newPowerState = OutOfBandManagement.PowerState.getStateMachine().getNextState(currentPowerState, event);
+ boolean result = outOfBandManagementDao.updateState(currentPowerState, event, newPowerState, outOfBandManagementHost, null);
+ if (result) {
+ final String message = String.format("Transitioned out-of-band management power state from:%s to:%s due to event:%s for the host id:%d", currentPowerState, newPowerState, event, outOfBandManagementHost.getHostId());
+ LOG.debug(message);
+ ActionEventUtils.onActionEvent(CallContext.current().getCallingUserId(), CallContext.current().getCallingAccountId(), Domain.ROOT_DOMAIN,
+ EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_POWERSTATE_TRANSITION, message);
+ }
+ return result;
+ } catch (NoTransitionException ignored) {
+ LOG.trace(String.format("Unable to transition out-of-band management power state for host id=%s for the event=%s and current power state=%s", outOfBandManagementHost.getHostId(), event, currentPowerState));
+ }
+ return false;
+ }
+
+ private boolean isOutOfBandManagementEnabledForZone(Long zoneId) {
+ if (zoneId == null) {
+ return true;
+ }
+ final DataCenterDetailVO zoneDetails = dataCenterDetailsDao.findDetail(zoneId, OOBM_ENABLED_DETAIL);
+ if (zoneDetails != null && !Strings.isNullOrEmpty(zoneDetails.getValue()) && !Boolean.valueOf(zoneDetails.getValue())) {
+ return false;
+ }
+ return true;
+ }
+
+ private boolean isOutOfBandManagementEnabledForCluster(Long clusterId) {
+ if (clusterId == null) {
+ return true;
+ }
+ final ClusterDetailsVO clusterDetails = clusterDetailsDao.findDetail(clusterId, OOBM_ENABLED_DETAIL);
+ if (clusterDetails != null && !Strings.isNullOrEmpty(clusterDetails.getValue()) && !Boolean.valueOf(clusterDetails.getValue())) {
+ return false;
+ }
+ return true;
+ }
+
+ private boolean isOutOfBandManagementEnabledForHost(Long hostId) {
+ if (hostId == null) {
+ return false;
+ }
+ final OutOfBandManagement outOfBandManagementConfig = outOfBandManagementDao.findByHost(hostId);
+ if (outOfBandManagementConfig == null || !outOfBandManagementConfig.isEnabled()) {
+ return false;
+ }
+ return true;
+ }
+
+ private void checkOutOfBandManagementEnabledByZoneClusterHost(final Host host) {
+ if (!isOutOfBandManagementEnabledForZone(host.getDataCenterId())) {
+ throw new CloudRuntimeException("Out-of-band management is disabled for the host's zone. Aborting Operation.");
+ }
+ if (!isOutOfBandManagementEnabledForCluster(host.getClusterId())) {
+ throw new CloudRuntimeException("Out-of-band management is disabled for the host's cluster. Aborting Operation.");
+ }
+ if (!isOutOfBandManagementEnabledForHost(host.getId())) {
+ throw new CloudRuntimeException("Out-of-band management is disabled or not configured for the host. Aborting Operation.");
+ }
+ }
+
+ public boolean isOutOfBandManagementEnabled(final Host host) {
+ return isOutOfBandManagementEnabledForZone(host.getDataCenterId())
+ && isOutOfBandManagementEnabledForCluster(host.getClusterId())
+ && isOutOfBandManagementEnabledForHost(host.getId());
+ }
+
+ public boolean transitionPowerStateToDisabled(List<? extends Host> hosts) {
+ boolean result = true;
+ for (Host host : hosts) {
+ result = result && transitionPowerState(OutOfBandManagement.PowerState.Event.Disabled,
+ outOfBandManagementDao.findByHost(host.getId()));
+ }
+ return result;
+ }
+
+ public void submitBackgroundPowerSyncTask(final Host host) {
+ if (host != null) {
+ // Note: This is a blocking queue based executor
+ backgroundSyncExecutor.submit(new OutOfBandManagementBackgroundTask(this, host, OutOfBandManagement.PowerOperation.STATUS));
+ }
+ }
+
+ private OutOfBandManagementResponse buildEnableDisableResponse(final boolean enabled) {
+ final OutOfBandManagementResponse response = new OutOfBandManagementResponse();
+ response.setEnabled(enabled);
+ response.setSuccess(true);
+ return response;
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_ENABLEDISABLE, eventDescription = "enabling/disabling out-of-band management on a zone")
+ public OutOfBandManagementResponse enableDisableOutOfBandManagement(final DataCenter zone, final boolean enabled) {
+ dataCenterDetailsDao.persist(zone.getId(), OOBM_ENABLED_DETAIL, String.valueOf(enabled));
+ if (!enabled) {
+ transitionPowerStateToDisabled(hostDao.findByDataCenterId(zone.getId()));
+ }
+ return buildEnableDisableResponse(enabled);
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_ENABLEDISABLE, eventDescription = "enabling/disabling out-of-band management on a cluster")
+ public OutOfBandManagementResponse enableDisableOutOfBandManagement(final Cluster cluster, final boolean enabled) {
+ clusterDetailsDao.persist(cluster.getId(), OOBM_ENABLED_DETAIL, String.valueOf(enabled));
+ if (!enabled) {
+ transitionPowerStateToDisabled(hostDao.findByClusterId(cluster.getId()));
+ }
+ return buildEnableDisableResponse(enabled);
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_ENABLEDISABLE, eventDescription = "enabling/disabling out-of-band management on a host")
+ public OutOfBandManagementResponse enableDisableOutOfBandManagement(final Host host, final boolean enabled) {
+ final OutOfBandManagement outOfBandManagementConfig = outOfBandManagementDao.findByHost(host.getId());
+ if (outOfBandManagementConfig == null) {
+ final OutOfBandManagementResponse response = new OutOfBandManagementResponse(null);
+ response.setSuccess(false);
+ response.setResultDescription("Out-of-band management is not configured for the host. Please configure the host before enabling/disabling it.");
+ return response;
+ }
+ hostAlertCache.invalidate(host.getId());
+ outOfBandManagementConfig.setEnabled(enabled);
+ boolean updateResult = outOfBandManagementDao.update(outOfBandManagementConfig.getId(), (OutOfBandManagementVO) outOfBandManagementConfig);
+ if (updateResult && !enabled) {
+ transitionPowerStateToDisabled(Collections.singletonList(host));
+ }
+ return buildEnableDisableResponse(enabled && updateResult);
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_CONFIGURE, eventDescription = "updating out-of-band management configuration")
+ public OutOfBandManagementResponse configureOutOfBandManagement(final Host host, final ImmutableMap<OutOfBandManagement.Option, String> options) {
+ OutOfBandManagement outOfBandManagementConfig = outOfBandManagementDao.findByHost(host.getId());
+ if (outOfBandManagementConfig == null) {
+ outOfBandManagementConfig = outOfBandManagementDao.persist(new OutOfBandManagementVO(host.getId()));
+ }
+ outOfBandManagementConfig = updateConfig(outOfBandManagementConfig, options);
+ if (Strings.isNullOrEmpty(outOfBandManagementConfig.getDriver()) || !outOfBandManagementDriversMap.containsKey(outOfBandManagementConfig.getDriver().toLowerCase())) {
+ throw new CloudRuntimeException("Out-of-band management driver is not available. Please provide a valid driver name.");
+ }
+
+ boolean updatedConfig = outOfBandManagementDao.update(outOfBandManagementConfig.getId(), (OutOfBandManagementVO) outOfBandManagementConfig);
+ CallContext.current().setEventDetails("host id:" + host.getId() + " configuration:" + outOfBandManagementConfig.getAddress() + ":" + outOfBandManagementConfig.getPort());
+
+ if (!updatedConfig) {
+ throw new CloudRuntimeException("Failed to update out-of-band management config for the host in the database.");
+ }
+
+ String result = "Out-of-band management successfully configured for the host";
+ LOG.debug(result);
+
+ final OutOfBandManagementResponse response = new OutOfBandManagementResponse(outOfBandManagementDao.findByHost(host.getId()));
+ response.setResultDescription(result);
+ response.setSuccess(true);
+ return response;
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_ACTION, eventDescription = "issuing Host out-of-band management action", async = true)
+ public OutOfBandManagementResponse executeOutOfBandManagementPowerOperation(final Host host, final OutOfBandManagement.PowerOperation powerOperation, final Long timeout) {
+ checkOutOfBandManagementEnabledByZoneClusterHost(host);
+ final OutOfBandManagement outOfBandManagementConfig = outOfBandManagementDao.findByHost(host.getId());
+ final ImmutableMap<OutOfBandManagement.Option, String> options = getOptions(outOfBandManagementConfig);
+ final OutOfBandManagementDriver driver = getDriver(outOfBandManagementConfig);
+
+ Long actionTimeOut = timeout;
+ if (actionTimeOut == null) {
+ actionTimeOut = OutOfBandManagementActionTimeout.valueIn(host.getClusterId());
+ }
+
+ final OutOfBandManagementDriverPowerCommand cmd = new OutOfBandManagementDriverPowerCommand(options, actionTimeOut, powerOperation);
+ final OutOfBandManagementDriverResponse driverResponse = driver.execute(cmd);
+
+ if (driverResponse == null) {
+ throw new CloudRuntimeException(String.format("Out-of-band Management action (%s) on host (%s) failed due to no response from the driver", powerOperation, host.getUuid()));
+ }
+
+ if (powerOperation.equals(OutOfBandManagement.PowerOperation.STATUS)) {
+ transitionPowerState(driverResponse.toEvent(), outOfBandManagementConfig);
+ }
+
+ if (!driverResponse.isSuccess()) {
+ String errorMessage = String.format("Out-of-band Management action (%s) on host (%s) failed with error: %s", powerOperation, host.getUuid(), driverResponse.getError());
+ if (driverResponse.hasAuthFailure()) {
+ errorMessage = String.format("Out-of-band Management action (%s) on host (%s) failed due to authentication error: %s. Please check configured credentials.", powerOperation, host.getUuid(), driverResponse.getError());
+ sendAuthError(host, errorMessage);
+ }
+ if (!powerOperation.equals(OutOfBandManagement.PowerOperation.STATUS)) {
+ LOG.debug(errorMessage);
+ }
+ throw new CloudRuntimeException(errorMessage);
+ }
+
+ final OutOfBandManagementResponse response = new OutOfBandManagementResponse(outOfBandManagementDao.findByHost(host.getId()));
+ response.setSuccess(driverResponse.isSuccess());
+ response.setResultDescription(driverResponse.getResult());
+ response.setId(host.getUuid());
+ response.setOutOfBandManagementAction(powerOperation.toString());
+ return response;
+ }
+
+ @Override
+ @ActionEvent(eventType = EventTypes.EVENT_HOST_OUTOFBAND_MANAGEMENT_CHANGE_PASSWORD, eventDescription = "updating out-of-band management password")
+ public OutOfBandManagementResponse changeOutOfBandManagementPassword(final Host host, final String newPassword) {
+ checkOutOfBandManagementEnabledByZoneClusterHost(host);
+ if (Strings.isNullOrEmpty(newPassword)) {
+ throw new CloudRuntimeException(String.format("Cannot change out-of-band management password as provided new-password is null or empty for the host %s.", host.getUuid()));
+ }
+ GlobalLock outOfBandManagementHostLock = GlobalLock.getInternLock(getOutOfBandManagementHostLock(host.getId()));
+ try {
+ if (outOfBandManagementHostLock.lock(ACQUIRE_GLOBAL_LOCK_TIMEOUT_FOR_HOST)) {
+ try {
+ final OutOfBandManagement outOfBandManagementConfig = outOfBandManagementDao.findByHost(host.getId());
+
+ final ImmutableMap<OutOfBandManagement.Option, String> options = getOptions(outOfBandManagementConfig);
+ if (!(options.containsKey(OutOfBandManagement.Option.PASSWORD) && !Strings.isNullOrEmpty(options.get(OutOfBandManagement.Option.PASSWORD)))) {
+ throw new CloudRuntimeException(String.format("Cannot change out-of-band management password as we've no previously configured password for the host %s.", host.getUuid()));
+ }
+ final OutOfBandManagementDriver driver = getDriver(outOfBandManagementConfig);
+
+ final OutOfBandManagementDriverChangePasswordCommand cmd = new OutOfBandManagementDriverChangePasswordCommand(options, OutOfBandManagementActionTimeout.valueIn(host.getClusterId()), newPassword);
+ final OutOfBandManagementDriverResponse driverResponse;
+ try {
+ driverResponse = driver.execute(cmd);
+ } catch (Exception e) {
+ LOG.error("Out-of-band management change password failed due to driver error: " + e.getMessage());
+ throw new CloudRuntimeException(String.format("Failed to change out-of-band management password for host (%s) due to driver error: %s", host.getUuid(), e.getMessage()));
+ }
+
+ if (!driverResponse.isSuccess()) {
+ throw new CloudRuntimeException(String.format("Failed to change out-of-band management password for host (%s) with error: %s", host.getUuid(), driverResponse.getError()));
+ }
+
+ final boolean updatedConfigResult = Transaction.execute(new TransactionCallback<Boolean>() {
+ @Override
+ public Boolean doInTransaction(TransactionStatus status) {
+ OutOfBandManagement updatedOutOfBandManagementConfig = outOfBandManagementDao.findByHost(host.getId());
+ updatedOutOfBandManagementConfig.setPassword(newPassword);
+ return outOfBandManagementDao.update(updatedOutOfBandManagementConfig.getId(), (OutOfBandManagementVO) updatedOutOfBandManagementConfig);
+ }
+ });
+
+ if (!updatedConfigResult) {
+ LOG.error(String.format("Succeeded to change out-of-band management password but failed to updated in database the new password:%s for the host id:%d", newPassword, host.getId()));
+ }
+
+ final OutOfBandManagementResponse response = new OutOfBandManagementResponse();
+ response.setSuccess(updatedConfigResult && driverResponse.isSuccess());
+ response.setResultDescription(driverResponse.getResult());
+ response.setId(host.getUuid());
+ return response;
+ } finally {
+ outOfBandManagementHostLock.unlock();
+ }
+ } else {
+ LOG.error("Unable to acquire synchronization lock to change out-of-band management password for host id: " + host.getId());
+ throw new CloudRuntimeException(String.format("Unable to acquire lock to change out-of-band management password for host (%s), please try after some time.", host.getUuid()));
+ }
+ } finally {
+ outOfBandManagementHostLock.releaseRef();
+ }
+ }
+
+ @Override
+ public String getName() {
+ return name;
+ }
+
+ @Override
+ public long getId() {
+ return serviceId;
+ }
+
+ @Override
+ public boolean configure(final String name, final Map<String, Object> params) throws ConfigurationException {
+ this.name = name;
+ this.serviceId = ManagementServerNode.getManagementServerId();
+
+ final int poolSize = OutOfBandManagementSyncThreadPoolSize.value();
+
+ hostAlertCache = CacheBuilder.newBuilder()
+ .concurrencyLevel(4)
+ .weakKeys()
+ .maximumSize(100 * poolSize)
+ .expireAfterWrite(1, TimeUnit.DAYS)
+ .build();
+
+ backgroundSyncExecutor = new ThreadPoolExecutor(poolSize, poolSize,
+ 0L, TimeUnit.MILLISECONDS,
+ new ArrayBlockingQueue<Runnable>(10 * poolSize, true), new ThreadPoolExecutor.CallerRunsPolicy());
+
+ LOG.info("Starting out-of-band management background sync executor with thread pool-size=" + poolSize + " and background sync thread interval=" + OutOfBandManagementSyncThreadInterval.value() + "s");
+ return true;
+ }
+
+ @Override
+ public boolean start() {
+ initializeDriversMap();
+ return true;
+ }
+
+ @Override
+ public boolean stop() {
+ backgroundSyncExecutor.shutdown();
+ outOfBandManagementDao.expireOutOfBandManagementOwnershipByServer(getId());
--- End diff --
When mgmt server loses db conn, it dies and in a clustered environment its peer will expire the ownership (see cluster/agent mgr impl).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by DaanHoogland <gi...@git.apache.org>.
Github user DaanHoogland commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-213391362
I did an one hour code review and found nothing that will make me :-1: it so; LGTM @swill I will however start integration tests and find more time as the size of the change warrants a two day review.
looks good @bhaisaab
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r61843006
--- Diff: api/src/org/apache/cloudstack/api/command/admin/outofbandmanagement/DisableOutOfBandManagementForClusterCmd.java ---
@@ -0,0 +1,105 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.api.command.admin.outofbandmanagement;
+
+import com.cloud.event.EventTypes;
+import com.cloud.exception.ConcurrentOperationException;
+import com.cloud.exception.InsufficientCapacityException;
+import com.cloud.exception.NetworkRuleConflictException;
+import com.cloud.exception.ResourceAllocationException;
+import com.cloud.exception.ResourceUnavailableException;
+import com.cloud.org.Cluster;
+import org.apache.cloudstack.acl.RoleType;
+import org.apache.cloudstack.api.APICommand;
+import org.apache.cloudstack.api.ApiConstants;
+import org.apache.cloudstack.api.ApiErrorCode;
+import org.apache.cloudstack.api.BaseAsyncCmd;
+import org.apache.cloudstack.api.BaseCmd;
+import org.apache.cloudstack.api.Parameter;
+import org.apache.cloudstack.api.ServerApiException;
+import org.apache.cloudstack.api.response.ClusterResponse;
+import org.apache.cloudstack.api.response.OutOfBandManagementResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.outofbandmanagement.OutOfBandManagementService;
+
+import javax.inject.Inject;
+
+@APICommand(name = "disableOutOfBandManagementForCluster", description = "Disables out-of-band management for a cluster",
--- End diff --
fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-217413162
@swill this PR is ready for CI test run and merge, thanks
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r62612293
--- Diff: utils/src/main/java/org/apache/cloudstack/utils/process/ProcessRunner.java ---
@@ -0,0 +1,112 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+
+package org.apache.cloudstack.utils.process;
+
+import com.cloud.utils.concurrency.NamedThreadFactory;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Strings;
+import org.apache.log4j.Logger;
+import org.joda.time.Duration;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.List;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+public class ProcessRunner {
+ public static final Logger LOG = Logger.getLogger(ProcessRunner.class);
+
+ private static final ExecutorService processExecutor = Executors.newCachedThreadPool(new NamedThreadFactory("ProcessRunner"));
+
+ private static String readStream(final InputStream inputStream) {
+ String text = null;
+ try {
+ final BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(inputStream));
+ String line;
+ while ((line = bufferedReader.readLine()) != null) {
+ if (Strings.isNullOrEmpty(text)) {
+ text = line;
+ } else {
+ text = text + "\n" + line;
+ }
+ }
+ } catch (IOException e) {
+ if (LOG.isTraceEnabled()) {
+ LOG.trace("ProcessRunner::readStream failed due to: " + e.getMessage());
+ }
+ }
+ return text;
+ }
+
+ public static ProcessResult executeCommands(final List<String> commands, final Duration timeOut) {
+ Preconditions.checkArgument(commands != null && timeOut != null);
+
+ int retVal = -2;
+ String stdOutput = null;
+ String stdError = null;
+
+ try {
+ final Process process = new ProcessBuilder().command(commands).start();
+ if (timeOut.getStandardSeconds() > 0) {
+ final Future<Integer> processFuture = processExecutor.submit(new Callable<Integer>() {
+ @Override
+ public Integer call() throws Exception {
+ return process.waitFor();
+ }
+ });
+ try {
+ retVal = processFuture.get(timeOut.getStandardSeconds(), TimeUnit.SECONDS);
+ } catch (ExecutionException | TimeoutException e) {
+ retVal = -1;
+ stdError = "Operation timed out, aborted";
+ if (LOG.isTraceEnabled()) {
+ LOG.trace("Failed to complete the requested command within timeout: " + e.getMessage());
+ }
+ } finally {
+ if (Strings.isNullOrEmpty(stdError)) {
+ stdOutput = readStream(process.getInputStream());
+ stdError = readStream(process.getErrorStream());
+ }
+ }
+ } else {
+ retVal = process.waitFor();
+ stdOutput = readStream(process.getInputStream());
+ stdError = readStream(process.getErrorStream());
+ }
+ process.destroy();
--- End diff --
process.destroy() is common to both cases when timeout is 0 or non-zero; cleanup is ensured in either case, as all exceptions are handled.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1502#discussion_r62321963
--- Diff: server/src/com/cloud/server/StatsCollector.java ---
@@ -412,6 +428,36 @@ protected void runInContext() {
}
}
+ class HostOutOfBandManagementStatsCollector extends ManagedContextRunnable {
+ @Override
+ protected void runInContext() {
+ try {
+ s_logger.debug("HostOutOfBandManagementStatsCollector is running...");
+ List<OutOfBandManagementVO> outOfBandManagementHosts = outOfBandManagementDao.findAllByManagementServer(ManagementServerNode.getManagementServerId());
+ if (outOfBandManagementHosts == null) {
+ return;
+ }
+ for (OutOfBandManagement outOfBandManagementHost : outOfBandManagementHosts) {
+ Host host = _hostDao.findById(outOfBandManagementHost.getHostId());
+ if (host == null) {
+ continue;
+ }
+ if (outOfBandManagementService.isOutOfBandManagementEnabled(host)) {
+ outOfBandManagementService.submitBackgroundPowerSyncTask(host);
+ } else {
+ if (outOfBandManagementHost.getPowerState() != OutOfBandManagement.PowerState.Disabled) {
+ if (outOfBandManagementService.transitionPowerStateToDisabled(Collections.singletonList(host))) {
--- End diff --
Collapse these two ``if`` blocks into a single ``else if`` on line 447 to improve readability.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by swill <gi...@git.apache.org>.
Github user swill commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-216884149
Please force push to kick off travis again. Thanks...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-218373527
@jburwell can you do a final review, LGTM or share further improvements. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by swill <gi...@git.apache.org>.
Github user swill commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-218875130
Jenkins for #1537 is also being held up by these tests:
```
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.apache.cloudstack.outofbandmanagement.driver.ipmitool.IpmitoolWrapperTest
Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 300.337 sec <<< FAILURE! - in org.apache.cloudstack.outofbandmanagement.driver.ipmitool.IpmitoolWrapperTest
testExecuteCommands(org.apache.cloudstack.outofbandmanagement.driver.ipmitool.IpmitoolWrapperTest) Time elapsed: 300.086 sec <<< FAILURE!
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at org.apache.cloudstack.outofbandmanagement.driver.ipmitool.IpmitoolWrapperTest.testExecuteCommands(IpmitoolWrapperTest.java:112)
Results :
Failed tests:
IpmitoolWrapperTest.testExecuteCommands:112 null
Tests run: 8, Failures: 1, Errors: 0, Skipped: 0
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9299: Out-of-band Management f...
Posted by swill <gi...@git.apache.org>.
Github user swill commented on the pull request:
https://github.com/apache/cloudstack/pull/1502#issuecomment-217606318
### CI RESULTS
```
Tests Run: 85
Skipped: 0
Failed: 1
Errors: 0
Duration: 6h 07m 49s
```
**Summary of the problem(s):**
```
FAIL: test_03_vpc_privategw_restart_vpc_cleanup (integration.smoke.test_privategw_acl.TestPrivateGwACL)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/data/git/cs1/cloudstack/test/integration/smoke/test_privategw_acl.py", line 265, in test_03_vpc_privategw_restart_vpc_cleanup
self.performVPCTests(vpc_off, True)
File "/data/git/cs1/cloudstack/test/integration/smoke/test_privategw_acl.py", line 332, in performVPCTests
self.check_pvt_gw_connectivity(vm2, public_ip_2, vm1.nic[0].ipaddress)
File "/data/git/cs1/cloudstack/test/integration/smoke/test_privategw_acl.py", line 559, in check_pvt_gw_connectivity
"Ping to outside world from VM should be successful"
AssertionError: Ping to outside world from VM should be successful
----------------------------------------------------------------------
Additional details in: /tmp/MarvinLogs/test_network_FIVJNA/results.txt
```
**Associated Uploads**
**`/tmp/MarvinLogs/DeployDataCenter__May_06_2016_21_31_06_LPI7VW:`**
* [dc_entries.obj](https://objects-east.cloud.ca/v1/e465abe2f9ae4478b9fff416eab61bd9/PR1502/tmp/MarvinLogs/DeployDataCenter__May_06_2016_21_31_06_LPI7VW/dc_entries.obj)
* [failed_plus_exceptions.txt](https://objects-east.cloud.ca/v1/e465abe2f9ae4478b9fff416eab61bd9/PR1502/tmp/MarvinLogs/DeployDataCenter__May_06_2016_21_31_06_LPI7VW/failed_plus_exceptions.txt)
* [runinfo.txt](https://objects-east.cloud.ca/v1/e465abe2f9ae4478b9fff416eab61bd9/PR1502/tmp/MarvinLogs/DeployDataCenter__May_06_2016_21_31_06_LPI7VW/runinfo.txt)
**`/tmp/MarvinLogs/test_network_FIVJNA:`**
* [failed_plus_exceptions.txt](https://objects-east.cloud.ca/v1/e465abe2f9ae4478b9fff416eab61bd9/PR1502/tmp/MarvinLogs/test_network_FIVJNA/failed_plus_exceptions.txt)
* [results.txt](https://objects-east.cloud.ca/v1/e465abe2f9ae4478b9fff416eab61bd9/PR1502/tmp/MarvinLogs/test_network_FIVJNA/results.txt)
* [runinfo.txt](https://objects-east.cloud.ca/v1/e465abe2f9ae4478b9fff416eab61bd9/PR1502/tmp/MarvinLogs/test_network_FIVJNA/runinfo.txt)
**`/tmp/MarvinLogs/test_vpc_routers_WUHEVI:`**
* [failed_plus_exceptions.txt](https://objects-east.cloud.ca/v1/e465abe2f9ae4478b9fff416eab61bd9/PR1502/tmp/MarvinLogs/test_vpc_routers_WUHEVI/failed_plus_exceptions.txt)
* [results.txt](https://objects-east.cloud.ca/v1/e465abe2f9ae4478b9fff416eab61bd9/PR1502/tmp/MarvinLogs/test_vpc_routers_WUHEVI/results.txt)
* [runinfo.txt](https://objects-east.cloud.ca/v1/e465abe2f9ae4478b9fff416eab61bd9/PR1502/tmp/MarvinLogs/test_vpc_routers_WUHEVI/runinfo.txt)
Uploads will be available until `2016-07-07 02:00:00 +0200 CEST`
*Comment created by [`upr comment`](https://github.com/cloudops/upr).*
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---