You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by mike-tutkowski <gi...@git.apache.org> on 2016/02/06 18:21:11 UTC
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
GitHub user mike-tutkowski opened a pull request:
https://github.com/apache/cloudstack/pull/1403
Taking fast and efficient volume snapshots with XenServer (and your storage provider)
A XenServer storage repository (SR) and virtual disk image (VDI) each have UUIDs that are immutable.
This poses a problem for SAN snapshots, if you intend on mounting the underlying snapshot SR alongside the source SR (duplicate UUIDs).
VMware has a solution for this called re-signaturing (so, in other words, the snapshot UUIDs can be changed).
This PR only deals with the CloudStack side of things, but it works in concert with a new XenServer storage manager created by CloudOps (this storage manager enables re-signaturing of XenServer SR and VDI UUIDs).
I have written Marvin integration tests to go along with this, but cannot yet check those into the CloudStack repo as they rely on SolidFire hardware.
If anyone would like to see these integration tests, please let me know.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mike-tutkowski/cloudstack xs-snapshots
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/cloudstack/pull/1403.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1403
----
commit 34dc4d7ca9cdc66fc16301d0f8a8ffef790e7462
Author: Mike Tutkowski <mi...@solidfire.com>
Date: 2015-11-16T19:18:25Z
Support for backend snapshots with XenServer
commit c0737879d7e0a5a8e991ab1f97c16db10fc79132
Author: Mike Tutkowski <mi...@solidfire.com>
Date: 2016-01-04T05:36:52Z
Initial changes to make use of both SolidFire snapshots and SolidFire cloning
commit bae4c2780d30194da6149b98248e6fcd0e4faa84
Author: Mike Tutkowski <mi...@solidfire.com>
Date: 2015-11-02T23:58:09Z
Refactoring and enhancements to SolidFire Integration-Testing API Plug-in
commit 8c551edca3fa7ebdc86e69fd747bb8a14d4ed178
Author: Mike Tutkowski <mi...@solidfire.com>
Date: 2016-01-05T08:32:39Z
Only "permanently" make use of cloning a volume from a snapshot when creating a new CloudStack volume from a volume snapshot
commit db3b66dfd6b172d7dd1a731abe200ef563f1439d
Author: Mike Tutkowski <mi...@solidfire.com>
Date: 2016-01-07T19:13:53Z
Access-related optimization for creating a template from a snapshot
commit a0a8da3b0044d1b388a8a0aa9e8f492b067eb807
Author: Mike Tutkowski <mi...@solidfire.com>
Date: 2016-01-07T19:16:48Z
Correction to used-space calculation
commit 054f110d47543c966a83cf797fea9a0d2046e6af
Author: Mike Tutkowski <mi...@solidfire.com>
Date: 2016-01-08T01:27:48Z
Do not check for remaining volume snapshots before deleting a volume snapshot that is supported by a back-end volume (only do this when deleting a volume snapshot that is supported by a back-end snapshot)
commit d8cf010117f664854c82b68a2d4a0d9b4b1e7c25
Author: Mike Tutkowski <mi...@solidfire.com>
Date: 2016-01-08T02:27:57Z
When deleting a CloudStack volume that is tied to a SolidFire volume, only delete the SolidFire volume if the CloudStack volume's volume snapshots (if any) are all supported by SolidFire volumes (as opposed to any being supported by SolidFire snapshots)
commit 8655fb1aa3a1cc6ee5443607836a74b103814b02
Author: Mike Tutkowski <mi...@solidfire.com>
Date: 2016-01-08T20:22:43Z
"=" should be "=="
commit 836c9a5b8ae6b2b8175166ba1418b9f59314cb4b
Author: Mike Tutkowski <mi...@solidfire.com>
Date: 2016-01-13T19:26:25Z
For integration-test purposes: Get snapshot details
commit 9528ea8d23ac41b7f5d9735ec29572781ee16e27
Author: Mike Tutkowski <mi...@solidfire.com>
Date: 2016-01-20T02:07:42Z
Enabling support for arbitrary key/value pairs to more easily be stored for SolidFire volumes
commit acf15ed6af5ff7090b2b71bef9f70a72f87cab48
Author: Mike Tutkowski <mi...@solidfire.com>
Date: 2016-01-20T22:57:32Z
Enable the use of arbitrary key/value pairs when creating a SolidFire volume via cloning and when creating a SolidFire snapshot
commit 1c9516ab27923caa845bf99a3a2eab406a9d7a6f
Author: Mike Tutkowski <mi...@solidfire.com>
Date: 2016-01-20T23:57:12Z
Improved exception handling
commit 7625e188d264633beaf30b1ca04779c1890d02f6
Author: Mike Tutkowski <mi...@solidfire.com>
Date: 2016-01-22T18:43:18Z
The way resigning metadata is invoked has changed. Call SR.create with type RELVMOISCSI. An exception should be thrown when the time would otherwise have come for the create functionality to attach the SR. Check if "success" is returned. If so, invoke SR.introduce; else, re-throw the exception.
commit a0fdf10246aa0a6f123b8583862322d30ecf0f38
Author: Mike Tutkowski <mi...@solidfire.com>
Date: 2016-01-29T19:32:26Z
Check if hostVO == null (not if hostId == null)
commit e851a40caf1392afc27a6583b7e1786cdf579af1
Author: Mike Tutkowski <mi...@solidfire.com>
Date: 2016-02-04T23:19:57Z
If the volume snapshot is backed by a SolidFire snapshot (as opposed to a SolidFire volume), then add it to the list.
commit 454f005ea3701b3ae47b8e0584eab658a331c5c0
Author: Mike Tutkowski <mi...@solidfire.com>
Date: 2016-02-06T03:58:52Z
Correcting an issue with a rebase
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r59082189
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -361,59 +608,225 @@ private String getProperty(long snapshotId, String property) {
}
private Map<String, String> getVolumeDetails(VolumeInfo volumeInfo) {
- Map<String, String> sourceDetails = new HashMap<String, String>();
+ Map<String, String> volumeDetails = new HashMap<String, String>();
VolumeVO volumeVO = _volumeDao.findById(volumeInfo.getId());
long storagePoolId = volumeVO.getPoolId();
StoragePoolVO storagePoolVO = _storagePoolDao.findById(storagePoolId);
- sourceDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
- sourceDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
- sourceDetails.put(DiskTO.IQN, volumeVO.get_iScsiName());
+ volumeDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
+ volumeDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
+ volumeDetails.put(DiskTO.IQN, volumeVO.get_iScsiName());
ChapInfo chapInfo = _volumeService.getChapInfo(volumeInfo, volumeInfo.getDataStore());
if (chapInfo != null) {
- sourceDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, chapInfo.getInitiatorUsername());
- sourceDetails.put(DiskTO.CHAP_INITIATOR_SECRET, chapInfo.getInitiatorSecret());
- sourceDetails.put(DiskTO.CHAP_TARGET_USERNAME, chapInfo.getTargetUsername());
- sourceDetails.put(DiskTO.CHAP_TARGET_SECRET, chapInfo.getTargetSecret());
+ volumeDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, chapInfo.getInitiatorUsername());
+ volumeDetails.put(DiskTO.CHAP_INITIATOR_SECRET, chapInfo.getInitiatorSecret());
+ volumeDetails.put(DiskTO.CHAP_TARGET_USERNAME, chapInfo.getTargetUsername());
+ volumeDetails.put(DiskTO.CHAP_TARGET_SECRET, chapInfo.getTargetSecret());
+ }
+
+ return volumeDetails;
+ }
+
+ private Map<String, String> getSnapshotDetails(SnapshotInfo snapshotInfo) {
+ Map<String, String> snapshotDetails = new HashMap<String, String>();
+
+ long storagePoolId = snapshotInfo.getDataStore().getId();
+ StoragePoolVO storagePoolVO = _storagePoolDao.findById(storagePoolId);
+
+ snapshotDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
+ snapshotDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
+
+ long snapshotId = snapshotInfo.getId();
+
+ snapshotDetails.put(DiskTO.IQN, getProperty(snapshotId, DiskTO.IQN));
+
+ snapshotDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, getProperty(snapshotId, DiskTO.CHAP_INITIATOR_USERNAME));
+ snapshotDetails.put(DiskTO.CHAP_INITIATOR_SECRET, getProperty(snapshotId, DiskTO.CHAP_INITIATOR_SECRET));
+ snapshotDetails.put(DiskTO.CHAP_TARGET_USERNAME, getProperty(snapshotId, DiskTO.CHAP_TARGET_USERNAME));
+ snapshotDetails.put(DiskTO.CHAP_TARGET_SECRET, getProperty(snapshotId, DiskTO.CHAP_TARGET_SECRET));
+
+ return snapshotDetails;
+ }
+
+ private HostVO getHost(SnapshotInfo snapshotInfo) {
+ HostVO hostVO = getHost(snapshotInfo.getDataCenterId(), true);
+
+ if (hostVO == null) {
+ hostVO = getHost(snapshotInfo.getDataCenterId(), false);
+
+ if (hostVO == null) {
+ throw new CloudRuntimeException("Unable to locate an applicable host");
+ }
}
- return sourceDetails;
+ return hostVO;
}
- public HostVO getHost(long dataStoreId) {
- StoragePoolVO storagePoolVO = _storagePoolDao.findById(dataStoreId);
+ private HostVO getHost(Long zoneId, boolean computeClusterMustSupportResign) {
+ if (zoneId == null) {
+ throw new CloudRuntimeException("Zone ID cannot be null.");
+ }
- List<? extends Cluster> clusters = _mgr.searchForClusters(storagePoolVO.getDataCenterId(), new Long(0), Long.MAX_VALUE, HypervisorType.XenServer.toString());
+ List<? extends Cluster> clusters = _mgr.searchForClusters(zoneId, new Long(0), Long.MAX_VALUE, HypervisorType.XenServer.toString());
if (clusters == null) {
- throw new CloudRuntimeException("Unable to locate an applicable cluster");
+ clusters = new ArrayList<>();
}
+ Collections.shuffle(clusters, new Random(System.nanoTime()));
+
+ clusters:
for (Cluster cluster : clusters) {
if (cluster.getAllocationState() == AllocationState.Enabled) {
List<HostVO> hosts = _hostDao.findByClusterId(cluster.getId());
if (hosts != null) {
+ Collections.shuffle(hosts, new Random(System.nanoTime()));
+
for (HostVO host : hosts) {
if (host.getResourceState() == ResourceState.Enabled) {
- return host;
+ if (computeClusterMustSupportResign) {
+ if (computeClusterSupportsResign(cluster.getId())) {
+ return host;
+ }
+ else {
+ // no other host in the cluster in question should be able to satisfy our requirements here, so move on to the next cluster
+ continue clusters;
+ }
+ }
+ else {
+ return host;
+ }
}
}
}
}
}
- throw new CloudRuntimeException("Unable to locate an applicable cluster");
+ return null;
}
@Override
public void copyAsync(Map<VolumeInfo, DataStore> volumeMap, VirtualMachineTO vmTo, Host srcHost, Host destHost, AsyncCompletionCallback<CopyCommandResult> callback) {
CopyCommandResult result = new CopyCommandResult(null, null);
+
result.setResult("Unsupported operation requested for copying data.");
+
callback.complete(result);
}
+
+ private Map<String, String> getDetails(DataObject dataObj) {
+ if (dataObj instanceof VolumeInfo) {
+ return getVolumeDetails((VolumeInfo)dataObj);
+ }
+ else if (dataObj instanceof SnapshotInfo) {
+ return getSnapshotDetails((SnapshotInfo)dataObj);
+ }
+
+ throw new CloudRuntimeException("'dataObj' must be of type 'VolumeInfo' or 'SnapshotInfo'.");
+ }
+
+ private CopyCmdAnswer performResignature(DataObject dataObj, HostVO hostVO) {
+ return performResignature(dataObj, hostVO, false);
+ }
+
+ private CopyCmdAnswer performResignature(DataObject dataObj, HostVO hostVO, boolean keepGrantedAccess) {
+ long storagePoolId = dataObj.getDataStore().getId();
+ DataStore dataStore = _dataStoreMgr.getDataStore(storagePoolId, DataStoreRole.Primary);
+
+ Map<String, String> details = getDetails(dataObj);
+
+ ResignatureCommand command = new ResignatureCommand(details);
+
+ ResignatureAnswer answer = null;
+
+ try {
+ _volumeService.grantAccess(dataObj, hostVO, dataStore);
+
+ answer = (ResignatureAnswer)_agentMgr.send(hostVO.getId(), command);
+ }
+ catch (Exception ex) {
+ keepGrantedAccess = false;
+
+ throw new CloudRuntimeException(ex.getMessage());
+ }
+ finally {
+ if (keepGrantedAccess == false) {
+ try {
+ _volumeService.revokeAccess(dataObj, hostVO, dataStore);
+ }
+ catch (Exception ex) {
+ s_logger.debug(ex.getMessage(), ex);
+ }
+ }
+ }
+
+ if (answer == null || !answer.getResult()) {
+ final String errMsg;
+
+ if (answer != null && answer.getDetails() != null && !answer.getDetails().isEmpty()) {
+ errMsg = answer.getDetails();
+ }
+ else {
+ errMsg = "Unable to perform resignature operation";
+ }
+
+ throw new CloudRuntimeException(errMsg);
+ }
+
+ VolumeObjectTO newVolume = new VolumeObjectTO();
+
+ newVolume.setSize(answer.getSize());
+ newVolume.setPath(answer.getPath());
+ newVolume.setFormat(answer.getFormat());
+
+ return new CopyCmdAnswer(newVolume);
+ }
+
+ private CopyCmdAnswer performCopyOfVdi(VolumeInfo volumeInfo, SnapshotInfo snapshotInfo, HostVO hostVO) {
+ String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
+ int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
+ CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), volumeInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+
+ CopyCmdAnswer copyCmdAnswer = null;
+
+ try {
+ _volumeService.grantAccess(snapshotInfo, hostVO, snapshotInfo.getDataStore());
+ _volumeService.grantAccess(volumeInfo, hostVO, volumeInfo.getDataStore());
+
+ Map<String, String> srcDetails = getSnapshotDetails(snapshotInfo);
+
+ copyCommand.setOptions(srcDetails);
+
+ Map<String, String> destDetails = getVolumeDetails(volumeInfo);
+
+ copyCommand.setOptions2(destDetails);
+
+ copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
+ }
+ catch (Exception ex) {
+ throw new CloudRuntimeException(ex.getMessage());
+ }
+ finally {
+ try {
+ _volumeService.revokeAccess(snapshotInfo, hostVO, snapshotInfo.getDataStore());
+ }
+ catch (Exception ex) {
+ s_logger.debug(ex.getMessage(), ex);
--- End diff --
Would an operator want to know if this rollback operation failed? It seems like it should be logged to ``WARN``. Also, please provide context information about the ``snapshotInfo`` and ``host`` to assist with operational debugging.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by mike-tutkowski <gi...@git.apache.org>.
Github user mike-tutkowski commented on the pull request:
https://github.com/apache/cloudstack/pull/1403#issuecomment-210072035
@swill This is pretty strange. The code that is failing to compile for you is actually not part of the codebase for ee53705:
https://github.com/mike-tutkowski/cloudstack/tree/xs-snapshots/plugins/api
All API plug-ins are in the location above and the one failing for you is not in there.
What do you think, Will?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r59084967
--- Diff: engine/storage/snapshot/src/org/apache/cloudstack/storage/snapshot/StorageSystemSnapshotStrategy.java ---
@@ -440,41 +526,28 @@ private void markAsBackedUp(SnapshotObject snapshotObj) {
@Override
public StrategyPriority canHandle(Snapshot snapshot, SnapshotOperation op) {
- long volumeId = snapshot.getVolumeId();
- VolumeVO volumeVO = _volumeDao.findById(volumeId);
if (SnapshotOperation.REVERT.equals(op)) {
- if (volumeVO != null && ImageFormat.QCOW2.equals(volumeVO.getFormat()))
- return StrategyPriority.DEFAULT;
- else
- return StrategyPriority.CANT_HANDLE;
+ return StrategyPriority.CANT_HANDLE;
}
- long storagePoolId;
+ long volumeId = snapshot.getVolumeId();
- if (volumeVO == null) {
- SnapshotDataStoreVO snapshotStore = _snapshotStoreDao.findBySnapshot(snapshot.getId(), DataStoreRole.Primary);
+ VolumeVO volumeVO = _volumeDao.findByIdIncludingRemoved(volumeId);
- if (snapshotStore != null) {
- storagePoolId = snapshotStore.getDataStoreId();
- }
- else {
- throw new CloudRuntimeException("Unable to determine the storage pool of the snapshot");
- }
- }
- else {
- storagePoolId = volumeVO.getPoolId();
- }
+ long storagePoolId = volumeVO.getPoolId();
DataStore dataStore = _dataStoreMgr.getDataStore(storagePoolId, DataStoreRole.Primary);
- Map<String, String> mapCapabilities = dataStore.getDriver().getCapabilities();
+ if (dataStore != null) {
+ Map<String, String> mapCapabilities = dataStore.getDriver().getCapabilities();
- if (mapCapabilities != null) {
- String value = mapCapabilities.get(DataStoreCapabilities.STORAGE_SYSTEM_SNAPSHOT.toString());
- Boolean supportsStorageSystemSnapshots = new Boolean(value);
+ if (mapCapabilities != null) {
+ String value = mapCapabilities.get(DataStoreCapabilities.STORAGE_SYSTEM_SNAPSHOT.toString());
+ Boolean supportsStorageSystemSnapshots = new Boolean(value);
--- End diff --
``new Boolean`` skips the constant pool -- placing unnecessary pressure on the garbage collector. Please use ``Boolean.valueOf`` instead.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r59083203
--- Diff: engine/storage/integration-test/test/org/apache/cloudstack/storage/test/FakePrimaryDataStoreDriver.java ---
@@ -90,7 +96,7 @@ public void makeTakeSnapshotSucceed(boolean success) {
}
@Override
- public void revertSnapshot(SnapshotInfo snapshot, AsyncCompletionCallback<CommandResult> callback) {
+ public void revertSnapshot(SnapshotInfo snapshotOnImageStore, SnapshotInfo snapshotOnPrimaryStore, AsyncCompletionCallback<CommandResult> callback) {
--- End diff --
This appears to be the default implementation from the IDE. I realize that it predates this patch, but should method do more? If not, can we strike the IDE generated comment to avoid future confusion?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by mike-tutkowski <gi...@git.apache.org>.
Github user mike-tutkowski commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r63078617
--- Diff: engine/storage/volume/src/org/apache/cloudstack/storage/volume/VolumeServiceImpl.java ---
@@ -554,6 +574,51 @@ protected Void managedCopyBaseImageCallback(AsyncCallbackDispatcher<VolumeServic
return null;
}
+ protected Void createManagedTemplateImageCallback(AsyncCallbackDispatcher<VolumeServiceImpl, CreateCmdResult> callback, CreateVolumeContext<CreateCmdResult> context) {
+ CreateCmdResult result = callback.getResult();
+ VolumeApiResult res = new VolumeApiResult(null);
+
+ res.setResult(result.getResult());
+
+ AsyncCallFuture<VolumeApiResult> future = context.getFuture();
+ DataObject templateOnPrimaryStoreObj = context.getVolume();
+
+ if (result.isSuccess()) {
+ ((TemplateObject)templateOnPrimaryStoreObj).setInstallPath(result.getPath());
+ templateOnPrimaryStoreObj.processEvent(Event.OperationSuccessed, result.getAnswer());
+ }
+ else {
+ templateOnPrimaryStoreObj.processEvent(Event.OperationFailed);
+ }
+
+ future.complete(res);
+
+ return null;
+ }
+
+ protected Void copyManagedTemplateCallback(AsyncCallbackDispatcher<VolumeServiceImpl, CopyCommandResult> callback, CreateBaseImageContext<VolumeApiResult> context) {
--- End diff --
Hi @DaanHoogland This is a legacy problem and I corrected it in #816. Now that #816 has been merged and I rebased against master, I have the updated code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/cloudstack/pull/1403
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by mike-tutkowski <gi...@git.apache.org>.
Github user mike-tutkowski commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r63079362
--- Diff: engine/storage/volume/src/org/apache/cloudstack/storage/volume/VolumeServiceImpl.java ---
@@ -554,6 +574,51 @@ protected Void managedCopyBaseImageCallback(AsyncCallbackDispatcher<VolumeServic
return null;
}
+ protected Void createManagedTemplateImageCallback(AsyncCallbackDispatcher<VolumeServiceImpl, CreateCmdResult> callback, CreateVolumeContext<CreateCmdResult> context) {
+ CreateCmdResult result = callback.getResult();
+ VolumeApiResult res = new VolumeApiResult(null);
+
+ res.setResult(result.getResult());
+
+ AsyncCallFuture<VolumeApiResult> future = context.getFuture();
+ DataObject templateOnPrimaryStoreObj = context.getVolume();
+
+ if (result.isSuccess()) {
+ ((TemplateObject)templateOnPrimaryStoreObj).setInstallPath(result.getPath());
+ templateOnPrimaryStoreObj.processEvent(Event.OperationSuccessed, result.getAnswer());
+ }
+ else {
+ templateOnPrimaryStoreObj.processEvent(Event.OperationFailed);
+ }
+
+ future.complete(res);
+
+ return null;
+ }
+
+ protected Void copyManagedTemplateCallback(AsyncCallbackDispatcher<VolumeServiceImpl, CopyCommandResult> callback, CreateBaseImageContext<VolumeApiResult> context) {
--- End diff --
Actually, @DaanHoogland, I was thinking of a different time when I corrected an existing "Void" problem like this.
The reason I wrote it this way was to conform to existing code style in the file (which was using Void (instead of void) for these types of methods and I wasn't sure why, so I followed the pattern to be on the safe side).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by mike-tutkowski <gi...@git.apache.org>.
Github user mike-tutkowski commented on the pull request:
https://github.com/apache/cloudstack/pull/1403#issuecomment-218874072
@DaanHoogland I had to put the "Void" return types back. It is used for AOP and won't compile with "void" for those two methods.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on the pull request:
https://github.com/apache/cloudstack/pull/1403#issuecomment-216222647
@mike-tutkowski thanks, please rebase against master, squash the changes to a single commit
tag:mergeready
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by swill <gi...@git.apache.org>.
Github user swill commented on the pull request:
https://github.com/apache/cloudstack/pull/1403#issuecomment-210085510
i will blow away my git repo and start from scratch and we will go from there. :) thx...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by swill <gi...@git.apache.org>.
Github user swill commented on the pull request:
https://github.com/apache/cloudstack/pull/1403#issuecomment-219702190
### CI RESULTS
```
Tests Run: 85
Skipped: 0
Failed: 0
Errors: 0
Duration: 4h 20m 17s
```
**Associated Uploads**
**`/tmp/MarvinLogs/DeployDataCenter__May_17_2016_00_15_14_6DMLAP:`**
* [dc_entries.obj](https://objects-east.cloud.ca/v1/e465abe2f9ae4478b9fff416eab61bd9/PR1403/tmp/MarvinLogs/DeployDataCenter__May_17_2016_00_15_14_6DMLAP/dc_entries.obj)
* [failed_plus_exceptions.txt](https://objects-east.cloud.ca/v1/e465abe2f9ae4478b9fff416eab61bd9/PR1403/tmp/MarvinLogs/DeployDataCenter__May_17_2016_00_15_14_6DMLAP/failed_plus_exceptions.txt)
* [runinfo.txt](https://objects-east.cloud.ca/v1/e465abe2f9ae4478b9fff416eab61bd9/PR1403/tmp/MarvinLogs/DeployDataCenter__May_17_2016_00_15_14_6DMLAP/runinfo.txt)
**`/tmp/MarvinLogs/test_network_CRQ5C2:`**
* [failed_plus_exceptions.txt](https://objects-east.cloud.ca/v1/e465abe2f9ae4478b9fff416eab61bd9/PR1403/tmp/MarvinLogs/test_network_CRQ5C2/failed_plus_exceptions.txt)
* [results.txt](https://objects-east.cloud.ca/v1/e465abe2f9ae4478b9fff416eab61bd9/PR1403/tmp/MarvinLogs/test_network_CRQ5C2/results.txt)
* [runinfo.txt](https://objects-east.cloud.ca/v1/e465abe2f9ae4478b9fff416eab61bd9/PR1403/tmp/MarvinLogs/test_network_CRQ5C2/runinfo.txt)
**`/tmp/MarvinLogs/test_vpc_routers_KUJ7NL:`**
* [failed_plus_exceptions.txt](https://objects-east.cloud.ca/v1/e465abe2f9ae4478b9fff416eab61bd9/PR1403/tmp/MarvinLogs/test_vpc_routers_KUJ7NL/failed_plus_exceptions.txt)
* [results.txt](https://objects-east.cloud.ca/v1/e465abe2f9ae4478b9fff416eab61bd9/PR1403/tmp/MarvinLogs/test_vpc_routers_KUJ7NL/results.txt)
* [runinfo.txt](https://objects-east.cloud.ca/v1/e465abe2f9ae4478b9fff416eab61bd9/PR1403/tmp/MarvinLogs/test_vpc_routers_KUJ7NL/runinfo.txt)
Uploads will be available until `2016-07-17 02:00:00 +0200 CEST`
*Comment created by [`upr comment`](https://github.com/cloudops/upr).*
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by swill <gi...@git.apache.org>.
Github user swill commented on the pull request:
https://github.com/apache/cloudstack/pull/1403#issuecomment-218805790
@mike-tutkowski can you rebase this PR as we now have merge conflicts. Can you review the recent comments and I will get this queued up to get CI run against it as soon as you have an updated version to make sure we are still good. Thanks...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r59079601
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -361,59 +608,225 @@ private String getProperty(long snapshotId, String property) {
}
private Map<String, String> getVolumeDetails(VolumeInfo volumeInfo) {
- Map<String, String> sourceDetails = new HashMap<String, String>();
+ Map<String, String> volumeDetails = new HashMap<String, String>();
VolumeVO volumeVO = _volumeDao.findById(volumeInfo.getId());
long storagePoolId = volumeVO.getPoolId();
StoragePoolVO storagePoolVO = _storagePoolDao.findById(storagePoolId);
- sourceDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
- sourceDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
- sourceDetails.put(DiskTO.IQN, volumeVO.get_iScsiName());
+ volumeDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
+ volumeDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
+ volumeDetails.put(DiskTO.IQN, volumeVO.get_iScsiName());
ChapInfo chapInfo = _volumeService.getChapInfo(volumeInfo, volumeInfo.getDataStore());
if (chapInfo != null) {
- sourceDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, chapInfo.getInitiatorUsername());
- sourceDetails.put(DiskTO.CHAP_INITIATOR_SECRET, chapInfo.getInitiatorSecret());
- sourceDetails.put(DiskTO.CHAP_TARGET_USERNAME, chapInfo.getTargetUsername());
- sourceDetails.put(DiskTO.CHAP_TARGET_SECRET, chapInfo.getTargetSecret());
+ volumeDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, chapInfo.getInitiatorUsername());
+ volumeDetails.put(DiskTO.CHAP_INITIATOR_SECRET, chapInfo.getInitiatorSecret());
+ volumeDetails.put(DiskTO.CHAP_TARGET_USERNAME, chapInfo.getTargetUsername());
+ volumeDetails.put(DiskTO.CHAP_TARGET_SECRET, chapInfo.getTargetSecret());
+ }
+
+ return volumeDetails;
+ }
+
+ private Map<String, String> getSnapshotDetails(SnapshotInfo snapshotInfo) {
+ Map<String, String> snapshotDetails = new HashMap<String, String>();
+
+ long storagePoolId = snapshotInfo.getDataStore().getId();
+ StoragePoolVO storagePoolVO = _storagePoolDao.findById(storagePoolId);
+
+ snapshotDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
+ snapshotDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
+
+ long snapshotId = snapshotInfo.getId();
+
+ snapshotDetails.put(DiskTO.IQN, getProperty(snapshotId, DiskTO.IQN));
+
+ snapshotDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, getProperty(snapshotId, DiskTO.CHAP_INITIATOR_USERNAME));
+ snapshotDetails.put(DiskTO.CHAP_INITIATOR_SECRET, getProperty(snapshotId, DiskTO.CHAP_INITIATOR_SECRET));
+ snapshotDetails.put(DiskTO.CHAP_TARGET_USERNAME, getProperty(snapshotId, DiskTO.CHAP_TARGET_USERNAME));
+ snapshotDetails.put(DiskTO.CHAP_TARGET_SECRET, getProperty(snapshotId, DiskTO.CHAP_TARGET_SECRET));
+
+ return snapshotDetails;
+ }
+
+ private HostVO getHost(SnapshotInfo snapshotInfo) {
+ HostVO hostVO = getHost(snapshotInfo.getDataCenterId(), true);
+
+ if (hostVO == null) {
+ hostVO = getHost(snapshotInfo.getDataCenterId(), false);
+
+ if (hostVO == null) {
+ throw new CloudRuntimeException("Unable to locate an applicable host");
+ }
}
- return sourceDetails;
+ return hostVO;
}
- public HostVO getHost(long dataStoreId) {
- StoragePoolVO storagePoolVO = _storagePoolDao.findById(dataStoreId);
+ private HostVO getHost(Long zoneId, boolean computeClusterMustSupportResign) {
+ if (zoneId == null) {
+ throw new CloudRuntimeException("Zone ID cannot be null.");
+ }
- List<? extends Cluster> clusters = _mgr.searchForClusters(storagePoolVO.getDataCenterId(), new Long(0), Long.MAX_VALUE, HypervisorType.XenServer.toString());
+ List<? extends Cluster> clusters = _mgr.searchForClusters(zoneId, new Long(0), Long.MAX_VALUE, HypervisorType.XenServer.toString());
if (clusters == null) {
- throw new CloudRuntimeException("Unable to locate an applicable cluster");
+ clusters = new ArrayList<>();
}
+ Collections.shuffle(clusters, new Random(System.nanoTime()));
+
+ clusters:
for (Cluster cluster : clusters) {
if (cluster.getAllocationState() == AllocationState.Enabled) {
List<HostVO> hosts = _hostDao.findByClusterId(cluster.getId());
if (hosts != null) {
+ Collections.shuffle(hosts, new Random(System.nanoTime()));
--- End diff --
See note above about avoiding seeding an RNG in a tight loop.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by mike-tutkowski <gi...@git.apache.org>.
Github user mike-tutkowski commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r63138267
--- Diff: engine/schema/src/com/cloud/dc/dao/ClusterDaoImpl.java ---
@@ -260,4 +268,41 @@ public boolean remove(Long id) {
sc.setParameters("dataCenterId", zoneId);
return customSearch(sc, null);
}
+
+ @Override
+ public boolean computeWhetherClusterSupportsResigning(long clusterId) {
+ ClusterVO cluster = findById(clusterId);
+
+ if (cluster == null || cluster.getAllocationState() != Grouping.AllocationState.Enabled) {
+ return false;
+ }
+
+ List<HostVO> hosts = hostDao.findByClusterId(clusterId);
+
+ if (hosts == null) {
+ return false;
+ }
+
+ for (HostVO host : hosts) {
+ if (host == null) {
+ return false;
+ }
+
+ DetailVO hostDetail = hostDetailsDao.findDetail(host.getId(), "supportsResign");
+
+ if (hostDetail == null) {
+ return false;
+ }
+
+ String value = hostDetail.getValue();
+
+ Boolean booleanValue = Boolean.valueOf(value);
+
+ if (booleanValue == false) {
+ return false;
+ }
+ }
+
+ return true;
+ }
--- End diff --
See what you think...I just added a method on the HostDetailsDao to return a Map where the key is the ID of the host and the value is true or false (you pass in the "name" field you are interested in).
This way we just do one query and then look up the results in the map each time through the "for" loop.
If we want to do a join of the host, host_details, and cluster tables (which, I believe, would be necessary to reduce the results coming back from the DB), perhaps you know of somewhere in our codebase where we do some similar action? Thanks
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by DaanHoogland <gi...@git.apache.org>.
Github user DaanHoogland commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r62880956
--- Diff: engine/storage/volume/src/org/apache/cloudstack/storage/volume/VolumeServiceImpl.java ---
@@ -134,6 +146,14 @@
EndPointSelector _epSelector;
@Inject
HostDao _hostDao;
+ @Inject
+ private PrimaryDataStoreDao _storagePoolDao;
--- End diff --
not used
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by mike-tutkowski <gi...@git.apache.org>.
Github user mike-tutkowski commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r63083245
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -361,59 +603,221 @@ private String getProperty(long snapshotId, String property) {
}
private Map<String, String> getVolumeDetails(VolumeInfo volumeInfo) {
- Map<String, String> sourceDetails = new HashMap<String, String>();
+ Map<String, String> volumeDetails = new HashMap<String, String>();
VolumeVO volumeVO = _volumeDao.findById(volumeInfo.getId());
long storagePoolId = volumeVO.getPoolId();
StoragePoolVO storagePoolVO = _storagePoolDao.findById(storagePoolId);
- sourceDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
- sourceDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
- sourceDetails.put(DiskTO.IQN, volumeVO.get_iScsiName());
+ volumeDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
+ volumeDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
+ volumeDetails.put(DiskTO.IQN, volumeVO.get_iScsiName());
ChapInfo chapInfo = _volumeService.getChapInfo(volumeInfo, volumeInfo.getDataStore());
if (chapInfo != null) {
- sourceDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, chapInfo.getInitiatorUsername());
- sourceDetails.put(DiskTO.CHAP_INITIATOR_SECRET, chapInfo.getInitiatorSecret());
- sourceDetails.put(DiskTO.CHAP_TARGET_USERNAME, chapInfo.getTargetUsername());
- sourceDetails.put(DiskTO.CHAP_TARGET_SECRET, chapInfo.getTargetSecret());
+ volumeDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, chapInfo.getInitiatorUsername());
+ volumeDetails.put(DiskTO.CHAP_INITIATOR_SECRET, chapInfo.getInitiatorSecret());
+ volumeDetails.put(DiskTO.CHAP_TARGET_USERNAME, chapInfo.getTargetUsername());
+ volumeDetails.put(DiskTO.CHAP_TARGET_SECRET, chapInfo.getTargetSecret());
}
- return sourceDetails;
+ return volumeDetails;
+ }
+
+ private Map<String, String> getSnapshotDetails(SnapshotInfo snapshotInfo) {
+ Map<String, String> snapshotDetails = new HashMap<String, String>();
+
+ long storagePoolId = snapshotInfo.getDataStore().getId();
+ StoragePoolVO storagePoolVO = _storagePoolDao.findById(storagePoolId);
+
+ snapshotDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
+ snapshotDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
+
+ long snapshotId = snapshotInfo.getId();
+
+ snapshotDetails.put(DiskTO.IQN, getProperty(snapshotId, DiskTO.IQN));
+
+ snapshotDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, getProperty(snapshotId, DiskTO.CHAP_INITIATOR_USERNAME));
+ snapshotDetails.put(DiskTO.CHAP_INITIATOR_SECRET, getProperty(snapshotId, DiskTO.CHAP_INITIATOR_SECRET));
+ snapshotDetails.put(DiskTO.CHAP_TARGET_USERNAME, getProperty(snapshotId, DiskTO.CHAP_TARGET_USERNAME));
+ snapshotDetails.put(DiskTO.CHAP_TARGET_SECRET, getProperty(snapshotId, DiskTO.CHAP_TARGET_SECRET));
+
+ return snapshotDetails;
}
- public HostVO getHost(long dataStoreId) {
- StoragePoolVO storagePoolVO = _storagePoolDao.findById(dataStoreId);
+ private HostVO getHost(SnapshotInfo snapshotInfo) {
+ HostVO hostVO = getHost(snapshotInfo.getDataCenterId(), true);
- List<? extends Cluster> clusters = _mgr.searchForClusters(storagePoolVO.getDataCenterId(), new Long(0), Long.MAX_VALUE, HypervisorType.XenServer.toString());
+ if (hostVO == null) {
+ hostVO = getHost(snapshotInfo.getDataCenterId(), false);
- if (clusters == null) {
- throw new CloudRuntimeException("Unable to locate an applicable cluster");
+ if (hostVO == null) {
+ throw new CloudRuntimeException("Unable to locate an applicable host in data center with ID = " + snapshotInfo.getDataCenterId());
+ }
}
- for (Cluster cluster : clusters) {
- if (cluster.getAllocationState() == AllocationState.Enabled) {
- List<HostVO> hosts = _hostDao.findByClusterId(cluster.getId());
+ return hostVO;
+ }
- if (hosts != null) {
- for (HostVO host : hosts) {
- if (host.getResourceState() == ResourceState.Enabled) {
- return host;
- }
- }
+ private HostVO getHost(Long zoneId, boolean computeClusterMustSupportResign) {
+ Preconditions.checkArgument(zoneId != null, "Zone ID cannot be null.");
+
+ List<HostVO> hosts = _hostDao.listByDataCenterIdAndHypervisorType(zoneId, HypervisorType.XenServer);
+
+ if (hosts == null) {
+ return null;
+ }
+
+ List<Long> clustersToSkip = new ArrayList<>();
+
+ Collections.shuffle(hosts, random);
+
+ for (HostVO host : hosts) {
+ if (computeClusterMustSupportResign) {
+ long clusterId = host.getClusterId();
+
+ if (clustersToSkip.contains(clusterId)) {
+ continue;
+ }
+
+ if (_clusterDao.computeClusterSupportsResign(clusterId)) {
+ return host;
+ }
+ else {
+ clustersToSkip.add(clusterId);
}
}
+ else {
+ return host;
+ }
}
- throw new CloudRuntimeException("Unable to locate an applicable cluster");
+ return null;
}
@Override
public void copyAsync(Map<VolumeInfo, DataStore> volumeMap, VirtualMachineTO vmTo, Host srcHost, Host destHost, AsyncCompletionCallback<CopyCommandResult> callback) {
CopyCommandResult result = new CopyCommandResult(null, null);
+
result.setResult("Unsupported operation requested for copying data.");
+
callback.complete(result);
}
+
+ private Map<String, String> getDetails(DataObject dataObj) {
+ if (dataObj instanceof VolumeInfo) {
+ return getVolumeDetails((VolumeInfo)dataObj);
+ }
+ else if (dataObj instanceof SnapshotInfo) {
+ return getSnapshotDetails((SnapshotInfo)dataObj);
+ }
+
+ throw new CloudRuntimeException("'dataObj' must be of type 'VolumeInfo' or 'SnapshotInfo'.");
+ }
+
+ private CopyCmdAnswer performResignature(DataObject dataObj, HostVO hostVO) {
+ return performResignature(dataObj, hostVO, false);
+ }
+
+ private CopyCmdAnswer performResignature(DataObject dataObj, HostVO hostVO, boolean keepGrantedAccess) {
--- End diff --
@DaanHoogland It's kind of like "Perform Re-signature Operation".
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by mike-tutkowski <gi...@git.apache.org>.
Github user mike-tutkowski commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r63132440
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -172,78 +258,207 @@ private void validate(SnapshotInfo snapshotInfo) {
}
}
- private Void handleCreateTemplateFromSnapshot(SnapshotInfo snapshotInfo, TemplateInfo templateInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
+ private boolean usingBackendSnapshotFor(SnapshotInfo snapshotInfo) {
+ String property = getProperty(snapshotInfo.getId(), "takeSnapshot");
+
+ return Boolean.parseBoolean(property);
+ }
+
+ private void handleCreateTemplateFromSnapshot(SnapshotInfo snapshotInfo, TemplateInfo templateInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
try {
snapshotInfo.processEvent(Event.CopyingRequested);
}
catch (Exception ex) {
throw new CloudRuntimeException("This snapshot is not currently in a state where it can be used to create a template.");
}
- HostVO hostVO = getHost(snapshotInfo.getDataStore().getId());
- DataStore srcDataStore = snapshotInfo.getDataStore();
+ HostVO hostVO = getHost(snapshotInfo);
- String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
- int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
- CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), templateInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+ boolean usingBackendSnapshot = usingBackendSnapshotFor(snapshotInfo);
+ boolean computeClusterSupportsResign = clusterDao.computeWhetherClusterSupportsResigning(hostVO.getClusterId());
- String errMsg = null;
-
- CopyCmdAnswer copyCmdAnswer = null;
+ if (usingBackendSnapshot && !computeClusterSupportsResign) {
+ throw new CloudRuntimeException("Unable to locate an applicable host with which to perform a resignature operation");
+ }
try {
- _volumeService.grantAccess(snapshotInfo, hostVO, srcDataStore);
+ if (usingBackendSnapshot) {
+ createVolumeFromSnapshot(hostVO, snapshotInfo, true);
+ }
- Map<String, String> srcDetails = getSnapshotDetails(_storagePoolDao.findById(srcDataStore.getId()), snapshotInfo);
+ DataStore srcDataStore = snapshotInfo.getDataStore();
- copyCommand.setOptions(srcDetails);
+ String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
+ int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
+ CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), templateInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+
+ String errMsg = null;
+
+ CopyCmdAnswer copyCmdAnswer = null;
- copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
- }
- catch (Exception ex) {
- throw new CloudRuntimeException(ex.getMessage());
- }
- finally {
try {
- _volumeService.revokeAccess(snapshotInfo, hostVO, srcDataStore);
+ // If we are using a back-end snapshot, then we should still have access to it from the hosts in the cluster that hostVO is in
+ // (because we passed in true as the third parameter to createVolumeFromSnapshot above).
+ if (usingBackendSnapshot == false) {
+ _volumeService.grantAccess(snapshotInfo, hostVO, srcDataStore);
+ }
+
+ Map<String, String> srcDetails = getSnapshotDetails(snapshotInfo);
+
+ copyCommand.setOptions(srcDetails);
+
+ copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
}
- catch (Exception ex) {
- s_logger.debug(ex.getMessage(), ex);
+ catch (AgentUnavailableException | OperationTimedoutException ex) {
+ throw new CloudRuntimeException("Failed to create template from snapshot : " + ex.getMessage());
}
+ finally {
+ try {
+ _volumeService.revokeAccess(snapshotInfo, hostVO, srcDataStore);
+ }
+ catch (Exception ex) {
+ s_logger.warn("error during revoking access to snapshot: " + ex.getMessage(), ex);
+ }
- if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
- if (copyCmdAnswer != null && copyCmdAnswer.getDetails() != null && !copyCmdAnswer.getDetails().isEmpty()) {
- errMsg = copyCmdAnswer.getDetails();
+ if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
+ if (copyCmdAnswer != null && !StringUtils.isEmpty(copyCmdAnswer.getDetails())) {
+ errMsg = copyCmdAnswer.getDetails();
+ }
+ else {
+ errMsg = "Unable to create template from snapshot";
+ }
}
- else {
- errMsg = "Unable to perform host-side operation";
+
+ try {
+ if (StringUtils.isEmpty(errMsg)) {
+ snapshotInfo.processEvent(Event.OperationSuccessed);
+ }
+ else {
+ snapshotInfo.processEvent(Event.OperationFailed);
+ }
+ }
+ catch (Exception ex) {
+ s_logger.warn("error while processing snapshot event" + ex.getMessage(), ex);
}
}
- try {
- if (errMsg == null) {
- snapshotInfo.processEvent(Event.OperationSuccessed);
+ CopyCommandResult result = new CopyCommandResult(null, copyCmdAnswer);
+
+ result.setResult(errMsg);
+
+ callback.complete(result);
+ }
+ finally {
+ if (usingBackendSnapshot) {
+ deleteVolumeFromSnapshot(snapshotInfo);
+ }
+ }
+ }
+
+ /**
+ * Clones a template present on the storage to a new volume and resignatures it.
+ *
+ * @param templateInfo source template
+ * @param volumeInfo destination ROOT volume
+ * @param callback for async
+ */
+ private void handleCreateVolumeFromTemplateBothOnStorageSystem(TemplateInfo templateInfo, VolumeInfo volumeInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
+ Preconditions.checkArgument(templateInfo != null, "Passing 'null' to templateInfo of handleCreateVolumeFromTemplateBothOnStorageSystem is not supported.");
+ Preconditions.checkArgument(volumeInfo != null, "Passing 'null' to volumeInfo of handleCreateVolumeFromTemplateBothOnStorageSystem is not supported.");
+
+ CopyCmdAnswer copyCmdAnswer = null;
+ String errMsg = null;
+
+ HostVO hostVO = getHost(volumeInfo.getDataCenterId(), true);
+
+ if (hostVO == null) {
+ throw new CloudRuntimeException("Unable to locate a host capable of resigning in the zone with the following ID: " + volumeInfo.getDataCenterId());
+ }
+
+ boolean computeClusterSupportsResign = clusterDao.computeWhetherClusterSupportsResigning(hostVO.getClusterId());
+
+ if (!computeClusterSupportsResign) {
+ throw new CloudRuntimeException("Unable to locate an applicable host with which to perform a resignature operation");
--- End diff --
Done
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by mike-tutkowski <gi...@git.apache.org>.
Github user mike-tutkowski commented on the pull request:
https://github.com/apache/cloudstack/pull/1403#issuecomment-217228903
@jburwell Just wanted to point out that an earlier PR of mine, #816, has four Marvin integration tests (three out of four of those apply to this PR, too). I wanted to mention that so you know they are going into the CloudStack repo (even though they require SolidFire hardware to run successfully). There are a few methods in the tests that I'd like to extract to a utils module, but I plan to do in the near future as I've been spending a lot more time building out Marvin tests in general lately. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by swill <gi...@git.apache.org>.
Github user swill commented on the pull request:
https://github.com/apache/cloudstack/pull/1403#issuecomment-218365435
@jburwell can I get your final review on this one? I think that is the only thing we are missing on this one. Thx...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by DaanHoogland <gi...@git.apache.org>.
Github user DaanHoogland commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r63083834
--- Diff: engine/storage/snapshot/src/org/apache/cloudstack/storage/snapshot/XenserverSnapshotStrategy.java ---
@@ -289,7 +291,7 @@ public boolean deleteSnapshot(Long snapshotId) {
@Override
public boolean revertSnapshot(SnapshotInfo snapshot) {
if (canHandle(snapshot,SnapshotOperation.REVERT) == StrategyPriority.CANT_HANDLE) {
- throw new UnsupportedOperationException("Reverting not supported. Create a template or volume based on the snapshot instead.");
+ throw new CloudRuntimeException("Reverting not supported. Create a template or volume based on the snapshot instead.");
--- End diff --
@jburwell want to discuss?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by mike-tutkowski <gi...@git.apache.org>.
Github user mike-tutkowski commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r59108724
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -114,6 +142,17 @@ private boolean canHandle(DataStore dataStore) {
return true;
}
+ } else if (dataObject instanceof TemplateInfo) {
+ // If the storage system can clone volumes, we can cache templates on it.
+ String value = mapCapabilities.get(DataStoreCapabilities.CAN_CREATE_VOLUME_FROM_VOLUME.toString());
--- End diff --
Yes, and value equaling null will get converted into a boolean of false later (only "true" (ignoring case) will result in a true boolean).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by swill <gi...@git.apache.org>.
Github user swill commented on the pull request:
https://github.com/apache/cloudstack/pull/1403#issuecomment-213603509
@DaanHoogland no, that is the next iteration. I just started on a smaller scale to try to get some better coverage. I will be working continually to increase the test coverage in bubble (expect some pull requests soon to bubble with more complete testing), as well as starting to try to improve testing in general by fixing tests that are currently failing. It is going to be a slow process, but I am slowly working on getting us to a place where we can have more confidence in the tests we are running.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by DaanHoogland <gi...@git.apache.org>.
Github user DaanHoogland commented on the pull request:
https://github.com/apache/cloudstack/pull/1403#issuecomment-213591446
h @swill, 120/120 passed :+1: so how did you decide on them? Are these the ones you send the 'here goes nothing' mail about?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r59081711
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -361,59 +608,225 @@ private String getProperty(long snapshotId, String property) {
}
private Map<String, String> getVolumeDetails(VolumeInfo volumeInfo) {
- Map<String, String> sourceDetails = new HashMap<String, String>();
+ Map<String, String> volumeDetails = new HashMap<String, String>();
VolumeVO volumeVO = _volumeDao.findById(volumeInfo.getId());
long storagePoolId = volumeVO.getPoolId();
StoragePoolVO storagePoolVO = _storagePoolDao.findById(storagePoolId);
- sourceDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
- sourceDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
- sourceDetails.put(DiskTO.IQN, volumeVO.get_iScsiName());
+ volumeDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
+ volumeDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
+ volumeDetails.put(DiskTO.IQN, volumeVO.get_iScsiName());
ChapInfo chapInfo = _volumeService.getChapInfo(volumeInfo, volumeInfo.getDataStore());
if (chapInfo != null) {
- sourceDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, chapInfo.getInitiatorUsername());
- sourceDetails.put(DiskTO.CHAP_INITIATOR_SECRET, chapInfo.getInitiatorSecret());
- sourceDetails.put(DiskTO.CHAP_TARGET_USERNAME, chapInfo.getTargetUsername());
- sourceDetails.put(DiskTO.CHAP_TARGET_SECRET, chapInfo.getTargetSecret());
+ volumeDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, chapInfo.getInitiatorUsername());
+ volumeDetails.put(DiskTO.CHAP_INITIATOR_SECRET, chapInfo.getInitiatorSecret());
+ volumeDetails.put(DiskTO.CHAP_TARGET_USERNAME, chapInfo.getTargetUsername());
+ volumeDetails.put(DiskTO.CHAP_TARGET_SECRET, chapInfo.getTargetSecret());
+ }
+
+ return volumeDetails;
+ }
+
+ private Map<String, String> getSnapshotDetails(SnapshotInfo snapshotInfo) {
+ Map<String, String> snapshotDetails = new HashMap<String, String>();
+
+ long storagePoolId = snapshotInfo.getDataStore().getId();
+ StoragePoolVO storagePoolVO = _storagePoolDao.findById(storagePoolId);
+
+ snapshotDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
+ snapshotDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
+
+ long snapshotId = snapshotInfo.getId();
+
+ snapshotDetails.put(DiskTO.IQN, getProperty(snapshotId, DiskTO.IQN));
+
+ snapshotDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, getProperty(snapshotId, DiskTO.CHAP_INITIATOR_USERNAME));
+ snapshotDetails.put(DiskTO.CHAP_INITIATOR_SECRET, getProperty(snapshotId, DiskTO.CHAP_INITIATOR_SECRET));
+ snapshotDetails.put(DiskTO.CHAP_TARGET_USERNAME, getProperty(snapshotId, DiskTO.CHAP_TARGET_USERNAME));
+ snapshotDetails.put(DiskTO.CHAP_TARGET_SECRET, getProperty(snapshotId, DiskTO.CHAP_TARGET_SECRET));
+
+ return snapshotDetails;
+ }
+
+ private HostVO getHost(SnapshotInfo snapshotInfo) {
+ HostVO hostVO = getHost(snapshotInfo.getDataCenterId(), true);
+
+ if (hostVO == null) {
+ hostVO = getHost(snapshotInfo.getDataCenterId(), false);
+
+ if (hostVO == null) {
+ throw new CloudRuntimeException("Unable to locate an applicable host");
+ }
}
- return sourceDetails;
+ return hostVO;
}
- public HostVO getHost(long dataStoreId) {
- StoragePoolVO storagePoolVO = _storagePoolDao.findById(dataStoreId);
+ private HostVO getHost(Long zoneId, boolean computeClusterMustSupportResign) {
+ if (zoneId == null) {
+ throw new CloudRuntimeException("Zone ID cannot be null.");
+ }
- List<? extends Cluster> clusters = _mgr.searchForClusters(storagePoolVO.getDataCenterId(), new Long(0), Long.MAX_VALUE, HypervisorType.XenServer.toString());
+ List<? extends Cluster> clusters = _mgr.searchForClusters(zoneId, new Long(0), Long.MAX_VALUE, HypervisorType.XenServer.toString());
if (clusters == null) {
- throw new CloudRuntimeException("Unable to locate an applicable cluster");
+ clusters = new ArrayList<>();
}
+ Collections.shuffle(clusters, new Random(System.nanoTime()));
+
+ clusters:
for (Cluster cluster : clusters) {
if (cluster.getAllocationState() == AllocationState.Enabled) {
List<HostVO> hosts = _hostDao.findByClusterId(cluster.getId());
if (hosts != null) {
+ Collections.shuffle(hosts, new Random(System.nanoTime()));
+
for (HostVO host : hosts) {
if (host.getResourceState() == ResourceState.Enabled) {
- return host;
+ if (computeClusterMustSupportResign) {
+ if (computeClusterSupportsResign(cluster.getId())) {
+ return host;
+ }
+ else {
+ // no other host in the cluster in question should be able to satisfy our requirements here, so move on to the next cluster
+ continue clusters;
+ }
+ }
+ else {
+ return host;
+ }
}
}
}
}
}
- throw new CloudRuntimeException("Unable to locate an applicable cluster");
+ return null;
}
@Override
public void copyAsync(Map<VolumeInfo, DataStore> volumeMap, VirtualMachineTO vmTo, Host srcHost, Host destHost, AsyncCompletionCallback<CopyCommandResult> callback) {
CopyCommandResult result = new CopyCommandResult(null, null);
+
result.setResult("Unsupported operation requested for copying data.");
+
callback.complete(result);
}
+
+ private Map<String, String> getDetails(DataObject dataObj) {
+ if (dataObj instanceof VolumeInfo) {
+ return getVolumeDetails((VolumeInfo)dataObj);
+ }
+ else if (dataObj instanceof SnapshotInfo) {
+ return getSnapshotDetails((SnapshotInfo)dataObj);
+ }
+
+ throw new CloudRuntimeException("'dataObj' must be of type 'VolumeInfo' or 'SnapshotInfo'.");
+ }
+
+ private CopyCmdAnswer performResignature(DataObject dataObj, HostVO hostVO) {
+ return performResignature(dataObj, hostVO, false);
+ }
+
+ private CopyCmdAnswer performResignature(DataObject dataObj, HostVO hostVO, boolean keepGrantedAccess) {
+ long storagePoolId = dataObj.getDataStore().getId();
+ DataStore dataStore = _dataStoreMgr.getDataStore(storagePoolId, DataStoreRole.Primary);
+
+ Map<String, String> details = getDetails(dataObj);
+
+ ResignatureCommand command = new ResignatureCommand(details);
+
+ ResignatureAnswer answer = null;
+
+ try {
+ _volumeService.grantAccess(dataObj, hostVO, dataStore);
+
+ answer = (ResignatureAnswer)_agentMgr.send(hostVO.getId(), command);
+ }
+ catch (Exception ex) {
+ keepGrantedAccess = false;
+
+ throw new CloudRuntimeException(ex.getMessage());
+ }
+ finally {
+ if (keepGrantedAccess == false) {
+ try {
+ _volumeService.revokeAccess(dataObj, hostVO, dataStore);
+ }
+ catch (Exception ex) {
+ s_logger.debug(ex.getMessage(), ex);
+ }
+ }
+ }
+
+ if (answer == null || !answer.getResult()) {
+ final String errMsg;
+
+ if (answer != null && answer.getDetails() != null && !answer.getDetails().isEmpty()) {
+ errMsg = answer.getDetails();
+ }
+ else {
+ errMsg = "Unable to perform resignature operation";
--- End diff --
Please add contextual information about the copy operation to the error message to assist with operational debugging.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by swill <gi...@git.apache.org>.
Github user swill commented on the pull request:
https://github.com/apache/cloudstack/pull/1403#issuecomment-207536244
Can you rebase and give me a squashed commit for this?
Also, regarding the code. I have seen a lot of people cleaning up code to remove the `_` in front of the variables. You seem to have done the opposite in this PR and have actually added an `_` to variables in some cases. I am not sure how we are handling this, but I suggest we start working toward a single standard as it is confusing for me right now...
I will add this to my queue to run a full CI against it again.
Also, I am looking for two distinct LGTM for every PR, so I would like another LGTM code review. Can someone from the dev list please give me a review? Thx...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by mike-tutkowski <gi...@git.apache.org>.
Github user mike-tutkowski commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r63086019
--- Diff: engine/storage/snapshot/src/org/apache/cloudstack/storage/snapshot/StorageSystemSnapshotStrategy.java ---
@@ -347,38 +395,87 @@ private String getProperty(long snapshotId, String property) {
return null;
}
- private HostVO getHost(Long hostId, VolumeVO volumeVO) {
- HostVO hostVO = _hostDao.findById(hostId);
+ private HostVO getHost(long volumeId) {
+ VolumeVO volumeVO = _volumeDao.findById(volumeId);
+
+ Long vmInstanceId = volumeVO.getInstanceId();
+ VMInstanceVO vmInstanceVO = _vmInstanceDao.findById(vmInstanceId);
+
+ Long hostId = null;
+
+ // if the volume to snapshot is associated with a VM
+ if (vmInstanceVO != null) {
+ hostId = vmInstanceVO.getHostId();
+
+ // if the VM is not associated with a host
+ if (hostId == null) {
+ hostId = vmInstanceVO.getLastHostId();
+ }
+ }
+
+ return getHost(volumeVO.getDataCenterId(), hostId);
+ }
+
+ private HostVO getHost(long zoneId, Long hostId) {
+ HostVO hostVO = getHost(zoneId, true);
if (hostVO != null) {
return hostVO;
}
- // pick a host in any XenServer cluster that's in the applicable zone
+ hostVO = _hostDao.findById(hostId);
- long zoneId = volumeVO.getDataCenterId();
+ if (hostVO != null) {
+ return hostVO;
+ }
+
+ hostVO = getHost(zoneId, false);
- List<? extends Cluster> clusters = _mgr.searchForClusters(zoneId, new Long(0), Long.MAX_VALUE, HypervisorType.XenServer.toString());
+ if (hostVO != null) {
+ return hostVO;
+ }
+
+ throw new CloudRuntimeException("Unable to locate an applicable host");
+ }
+
+ private HostVO getHost(long zoneId, boolean computeClusterMustSupportResign) {
+ List<? extends Cluster> clusters = _mgr.searchForClusters(zoneId, 0L, Long.MAX_VALUE, HypervisorType.XenServer.toString());
if (clusters == null) {
- throw new CloudRuntimeException("Unable to locate an applicable cluster");
+ clusters = new ArrayList<>();
}
+ Collections.shuffle(clusters, new Random(System.nanoTime()));
+
+ clusters:
for (Cluster cluster : clusters) {
if (cluster.getAllocationState() == AllocationState.Enabled) {
List<HostVO> hosts = _hostDao.findByClusterId(cluster.getId());
if (hosts != null) {
+ Collections.shuffle(hosts, new Random(System.nanoTime()));
+
for (HostVO host : hosts) {
if (host.getResourceState() == ResourceState.Enabled) {
- return host;
+ if (computeClusterMustSupportResign) {
+ if (_clusterDao.computeClusterSupportsResign(cluster.getId())) {
+ return host;
+ }
+ else {
+ // no other host in the cluster in question should be able to satisfy our requirements here, so move on to the next cluster
+ continue clusters;
+ }
+ }
+ else {
+ return host;
+ }
}
}
}
}
}
- throw new CloudRuntimeException("Unable to locate an applicable cluster");
+ return null;
--- End diff --
@DaanHoogland I changed this to make use of Guava's Optional class.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by mike-tutkowski <gi...@git.apache.org>.
Github user mike-tutkowski commented on the pull request:
https://github.com/apache/cloudstack/pull/1403#issuecomment-210084608
@swill I think I might know what happened.
There used to be a ApiSolidFireServiceImpl.java, but I renamed it in 940d5b.
It seems the file is likely still sitting on your filesystem and causing you this trouble.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by mike-tutkowski <gi...@git.apache.org>.
Github user mike-tutkowski commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r59110675
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -255,99 +448,153 @@ private Void handleCreateVolumeFromSnapshotBothOnStorageSystem(SnapshotInfo snap
VolumeApiResult result = future.get();
+ if (volumeDetail != null) {
+ _volumeDetailsDao.remove(volumeDetail.getId());
+ }
+
if (result.isFailed()) {
s_logger.debug("Failed to create a volume: " + result.getResult());
throw new CloudRuntimeException(result.getResult());
}
- }
- catch (Exception ex) {
- throw new CloudRuntimeException(ex.getMessage());
- }
- volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
+ volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
- volumeInfo.processEvent(Event.MigrationRequested);
+ volumeInfo.processEvent(Event.MigrationRequested);
- volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
+ volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
- HostVO hostVO = getHost(snapshotInfo.getDataStore().getId());
+ if (useCloning) {
+ copyCmdAnswer = performResignature(volumeInfo, hostVO);
+ }
+ else {
+ // asking for a XenServer host here so we don't always prefer to use XenServer hosts that support resigning
+ // even when we don't need those hosts to do this kind of copy work
+ hostVO = getHost(snapshotInfo.getDataCenterId(), false);
- String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
- int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
- CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), volumeInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+ copyCmdAnswer = performCopyOfVdi(volumeInfo, snapshotInfo, hostVO);
+ }
- CopyCmdAnswer copyCmdAnswer = null;
+ if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
+ if (copyCmdAnswer != null && !StringUtils.isEmpty(copyCmdAnswer.getDetails())) {
+ errMsg = copyCmdAnswer.getDetails();
+ }
+ else {
+ errMsg = "Unable to perform host-side operation";
+ }
+ }
+ }
+ catch (Exception ex) {
+ errMsg = ex.getMessage() != null ? ex.getMessage() : "Copy operation failed";
+ }
- try {
- _volumeService.grantAccess(snapshotInfo, hostVO, snapshotInfo.getDataStore());
- _volumeService.grantAccess(volumeInfo, hostVO, volumeInfo.getDataStore());
+ CopyCommandResult result = new CopyCommandResult(null, copyCmdAnswer);
- Map<String, String> srcDetails = getSnapshotDetails(_storagePoolDao.findById(snapshotInfo.getDataStore().getId()), snapshotInfo);
+ result.setResult(errMsg);
- copyCommand.setOptions(srcDetails);
+ callback.complete(result);
+ }
- Map<String, String> destDetails = getVolumeDetails(volumeInfo);
+ /**
+ * If the underlying storage system is making use of read-only snapshots, this gives the storage system the opportunity to
+ * create a volume from the snapshot so that we can copy the VHD file that should be inside of the snapshot to secondary storage.
+ *
+ * The resultant volume must be writable because we need to resign the SR and the VDI that should be inside of it before we copy
+ * the VHD file to secondary storage.
+ *
+ * If the storage system is using writable snapshots, then nothing need be done by that storage system here because we can just
+ * resign the SR and the VDI that should be inside of the snapshot before copying the VHD file to secondary storage.
+ */
+ private void createVolumeFromSnapshot(HostVO hostVO, SnapshotInfo snapshotInfo, boolean keepGrantedAccess) {
+ SnapshotDetailsVO snapshotDetails = handleSnapshotDetails(snapshotInfo.getId(), "tempVolume", "create");
- copyCommand.setOptions2(destDetails);
-
- copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
- }
- catch (Exception ex) {
- throw new CloudRuntimeException(ex.getMessage());
+ try {
+ snapshotInfo.getDataStore().getDriver().createAsync(snapshotInfo.getDataStore(), snapshotInfo, null);
}
finally {
- try {
- _volumeService.revokeAccess(snapshotInfo, hostVO, snapshotInfo.getDataStore());
- }
- catch (Exception ex) {
- s_logger.debug(ex.getMessage(), ex);
- }
-
- try {
- _volumeService.revokeAccess(volumeInfo, hostVO, volumeInfo.getDataStore());
- }
- catch (Exception ex) {
- s_logger.debug(ex.getMessage(), ex);
- }
+ _snapshotDetailsDao.remove(snapshotDetails.getId());
}
- String errMsg = null;
+ CopyCmdAnswer copyCmdAnswer = performResignature(snapshotInfo, hostVO, keepGrantedAccess);
if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
- if (copyCmdAnswer != null && copyCmdAnswer.getDetails() != null && !copyCmdAnswer.getDetails().isEmpty()) {
- errMsg = copyCmdAnswer.getDetails();
+ if (copyCmdAnswer != null && !StringUtils.isEmpty(copyCmdAnswer.getDetails())) {
+ throw new CloudRuntimeException(copyCmdAnswer.getDetails());
}
else {
- errMsg = "Unable to perform host-side operation";
+ throw new CloudRuntimeException("Unable to perform host-side operation");
}
}
+ }
- CopyCommandResult result = new CopyCommandResult(null, copyCmdAnswer);
+ /**
+ * If the underlying storage system needed to create a volume from a snapshot for createVolumeFromSnapshot(HostVO, SnapshotInfo), then
+ * this is its opportunity to delete that temporary volume and restore properties in snapshot_details to the way they were before the
+ * invocation of createVolumeFromSnapshot(HostVO, SnapshotInfo).
+ */
+ private void deleteVolumeFromSnapshot(SnapshotInfo snapshotInfo) {
+ SnapshotDetailsVO snapshotDetails = handleSnapshotDetails(snapshotInfo.getId(), "tempVolume", "delete");
- result.setResult(errMsg);
+ try {
+ snapshotInfo.getDataStore().getDriver().createAsync(snapshotInfo.getDataStore(), snapshotInfo, null);
+ }
+ finally {
+ _snapshotDetailsDao.remove(snapshotDetails.getId());
+ }
+ }
- callback.complete(result);
+ private SnapshotDetailsVO handleSnapshotDetails(long csSnapshotId, String name, String value) {
+ _snapshotDetailsDao.removeDetail(csSnapshotId, name);
- return null;
+ SnapshotDetailsVO snapshotDetails = new SnapshotDetailsVO(csSnapshotId, name, value, false);
+
+ return _snapshotDetailsDao.persist(snapshotDetails);
}
- private Map<String, String> getSnapshotDetails(StoragePoolVO storagePoolVO, SnapshotInfo snapshotInfo) {
- Map<String, String> details = new HashMap<String, String>();
+ private boolean canStorageSystemCreateVolumeFromVolume(SnapshotInfo snapshotInfo) {
+ boolean supportsCloningVolumeFromVolume = false;
- details.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
- details.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
+ DataStore dataStore = _dataStoreMgr.getDataStore(snapshotInfo.getDataStore().getId(), DataStoreRole.Primary);
- long snapshotId = snapshotInfo.getId();
+ Map<String, String> mapCapabilities = dataStore.getDriver().getCapabilities();
- details.put(DiskTO.IQN, getProperty(snapshotId, DiskTO.IQN));
+ if (mapCapabilities != null) {
+ String value = mapCapabilities.get(DataStoreCapabilities.CAN_CREATE_VOLUME_FROM_VOLUME.toString());
- details.put(DiskTO.CHAP_INITIATOR_USERNAME, getProperty(snapshotId, DiskTO.CHAP_INITIATOR_USERNAME));
- details.put(DiskTO.CHAP_INITIATOR_SECRET, getProperty(snapshotId, DiskTO.CHAP_INITIATOR_SECRET));
- details.put(DiskTO.CHAP_TARGET_USERNAME, getProperty(snapshotId, DiskTO.CHAP_TARGET_USERNAME));
- details.put(DiskTO.CHAP_TARGET_SECRET, getProperty(snapshotId, DiskTO.CHAP_TARGET_SECRET));
+ supportsCloningVolumeFromVolume = new Boolean(value);
+ }
- return details;
+ return supportsCloningVolumeFromVolume;
+ }
+
+ private boolean computeClusterSupportsResign(long clusterId) {
+ List<HostVO> hosts = _hostDao.findByClusterId(clusterId);
+
+ if (hosts == null) {
+ return false;
+ }
+
+ for (HostVO host : hosts) {
+ if (host == null) {
+ return false;
+ }
+
+ DetailVO hostDetail = _hostDetailsDao.findDetail(host.getId(), "supportsResign");
+
+ if (hostDetail == null) {
+ return false;
+ }
+
+ String value = hostDetail.getValue();
+
+ Boolean booleanValue = Boolean.valueOf(value);
+
+ if (booleanValue == false) {
+ return false;
+ }
+ }
--- End diff --
Are you thinking a new SQL statement like this?
Select count(*) = (Select count(*) From cloud.host Where cluster_id = ?)
From cloud.host_details
Where name = "supportsResign"
and value = "true"
and host_id in (Select id From cloud.host Where cluster_id = ?)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by mike-tutkowski <gi...@git.apache.org>.
Github user mike-tutkowski commented on the pull request:
https://github.com/apache/cloudstack/pull/1403#issuecomment-207559800
I'm not sure what the standard for "idiomatic" is here. :) I've pretty much usually seen "_" (and sometimes "m_") used to note a member variable versus a local variable. That is then used in concert with camel case.
Also, for static member variables that are public, I usually have seen all upper case with individual words separated by an underscore (but the variable not prefixed with an underscore).
We should probably develop a community standard.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by mike-tutkowski <gi...@git.apache.org>.
Github user mike-tutkowski commented on the pull request:
https://github.com/apache/cloudstack/pull/1403#issuecomment-210066399
@swill Weird, I've compiled ee53705 with the following command on two separate machines and I ran regression tests against that SHA for a few hours yesterday:
mvn -P developer,systemvm clean install -D noredist
Let me look into this - thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by DaanHoogland <gi...@git.apache.org>.
Github user DaanHoogland commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r62879575
--- Diff: engine/storage/snapshot/src/org/apache/cloudstack/storage/snapshot/StorageSystemSnapshotStrategy.java ---
@@ -347,38 +395,87 @@ private String getProperty(long snapshotId, String property) {
return null;
}
- private HostVO getHost(Long hostId, VolumeVO volumeVO) {
- HostVO hostVO = _hostDao.findById(hostId);
+ private HostVO getHost(long volumeId) {
+ VolumeVO volumeVO = _volumeDao.findById(volumeId);
+
+ Long vmInstanceId = volumeVO.getInstanceId();
+ VMInstanceVO vmInstanceVO = _vmInstanceDao.findById(vmInstanceId);
+
+ Long hostId = null;
+
+ // if the volume to snapshot is associated with a VM
+ if (vmInstanceVO != null) {
+ hostId = vmInstanceVO.getHostId();
+
+ // if the VM is not associated with a host
+ if (hostId == null) {
+ hostId = vmInstanceVO.getLastHostId();
+ }
+ }
+
+ return getHost(volumeVO.getDataCenterId(), hostId);
+ }
+
+ private HostVO getHost(long zoneId, Long hostId) {
+ HostVO hostVO = getHost(zoneId, true);
if (hostVO != null) {
return hostVO;
}
- // pick a host in any XenServer cluster that's in the applicable zone
+ hostVO = _hostDao.findById(hostId);
- long zoneId = volumeVO.getDataCenterId();
+ if (hostVO != null) {
+ return hostVO;
+ }
+
+ hostVO = getHost(zoneId, false);
- List<? extends Cluster> clusters = _mgr.searchForClusters(zoneId, new Long(0), Long.MAX_VALUE, HypervisorType.XenServer.toString());
+ if (hostVO != null) {
+ return hostVO;
+ }
+
+ throw new CloudRuntimeException("Unable to locate an applicable host");
+ }
+
+ private HostVO getHost(long zoneId, boolean computeClusterMustSupportResign) {
+ List<? extends Cluster> clusters = _mgr.searchForClusters(zoneId, 0L, Long.MAX_VALUE, HypervisorType.XenServer.toString());
if (clusters == null) {
- throw new CloudRuntimeException("Unable to locate an applicable cluster");
+ clusters = new ArrayList<>();
}
+ Collections.shuffle(clusters, new Random(System.nanoTime()));
+
+ clusters:
for (Cluster cluster : clusters) {
if (cluster.getAllocationState() == AllocationState.Enabled) {
List<HostVO> hosts = _hostDao.findByClusterId(cluster.getId());
if (hosts != null) {
+ Collections.shuffle(hosts, new Random(System.nanoTime()));
+
for (HostVO host : hosts) {
if (host.getResourceState() == ResourceState.Enabled) {
- return host;
+ if (computeClusterMustSupportResign) {
+ if (_clusterDao.computeClusterSupportsResign(cluster.getId())) {
+ return host;
+ }
+ else {
+ // no other host in the cluster in question should be able to satisfy our requirements here, so move on to the next cluster
--- End diff --
a usefull comment! kudos, that rarely happens ;)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by DaanHoogland <gi...@git.apache.org>.
Github user DaanHoogland commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r62886063
--- Diff: plugins/hypervisors/xenserver/src/com/cloud/hypervisor/xenserver/resource/CitrixResourceBase.java ---
@@ -1792,10 +1796,26 @@ protected void fillHostInfo(final Connection conn, final StartupRoutingCommand c
cmd.setPod(_pod);
cmd.setVersion(CitrixResourceBase.class.getPackage().getImplementationVersion());
+ try {
+ final String cmdLine = "xe sm-list | grep \"resigning of duplicates\"";
+
+ final XenServerUtilitiesHelper xenServerUtilitiesHelper = getXenServerUtilitiesHelper();
+
+ Pair<Boolean, String> result = xenServerUtilitiesHelper.executeSshWrapper(_host.getIp(), 22, _username, null, getPwdFromQueue(), cmdLine);
+
+ boolean supportsClonedVolumes = result != null && result.first() != null && result.first() &&
+ result.second() != null && result.second().length() > 0;
+
+ cmd.setSupportsClonedVolumes(supportsClonedVolumes);
+ } catch (Exception ex) {
--- End diff --
no checked exceptions are being thrown, why a catch all? is there a specific exception that might threaten us?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on the pull request:
https://github.com/apache/cloudstack/pull/1403#issuecomment-219148309
@DaanHoogland in my experience, stack traces are a critical piece of information for operationally debugging CloudStack. Unfortunately, our logging lacks the clarity and specify to use the error message alone. I hope our logging improves overtime to omit them from ``INFO``, ``WARN``, and ``ERROR``. However, today, this information is vital to resolving issues.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by DaanHoogland <gi...@git.apache.org>.
Github user DaanHoogland commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r62878912
--- Diff: engine/storage/snapshot/src/org/apache/cloudstack/storage/snapshot/StorageSystemSnapshotStrategy.java ---
@@ -72,8 +77,10 @@
private static final Logger s_logger = Logger.getLogger(StorageSystemSnapshotStrategy.class);
@Inject private AgentManager _agentMgr;
+ @Inject private ClusterDao _clusterDao;
@Inject private DataStoreManager _dataStoreMgr;
@Inject private HostDao _hostDao;
+ @Inject private HostDetailsDao _hostDetailsDao;
--- End diff --
warning for not used
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by DaanHoogland <gi...@git.apache.org>.
Github user DaanHoogland commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r62880464
--- Diff: engine/storage/snapshot/src/org/apache/cloudstack/storage/snapshot/XenserverSnapshotStrategy.java ---
@@ -289,7 +291,7 @@ public boolean deleteSnapshot(Long snapshotId) {
@Override
public boolean revertSnapshot(SnapshotInfo snapshot) {
if (canHandle(snapshot,SnapshotOperation.REVERT) == StrategyPriority.CANT_HANDLE) {
- throw new UnsupportedOperationException("Reverting not supported. Create a template or volume based on the snapshot instead.");
+ throw new CloudRuntimeException("Reverting not supported. Create a template or volume based on the snapshot instead.");
--- End diff --
why change the type of the exception?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r59082240
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -361,59 +608,225 @@ private String getProperty(long snapshotId, String property) {
}
private Map<String, String> getVolumeDetails(VolumeInfo volumeInfo) {
- Map<String, String> sourceDetails = new HashMap<String, String>();
+ Map<String, String> volumeDetails = new HashMap<String, String>();
VolumeVO volumeVO = _volumeDao.findById(volumeInfo.getId());
long storagePoolId = volumeVO.getPoolId();
StoragePoolVO storagePoolVO = _storagePoolDao.findById(storagePoolId);
- sourceDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
- sourceDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
- sourceDetails.put(DiskTO.IQN, volumeVO.get_iScsiName());
+ volumeDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
+ volumeDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
+ volumeDetails.put(DiskTO.IQN, volumeVO.get_iScsiName());
ChapInfo chapInfo = _volumeService.getChapInfo(volumeInfo, volumeInfo.getDataStore());
if (chapInfo != null) {
- sourceDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, chapInfo.getInitiatorUsername());
- sourceDetails.put(DiskTO.CHAP_INITIATOR_SECRET, chapInfo.getInitiatorSecret());
- sourceDetails.put(DiskTO.CHAP_TARGET_USERNAME, chapInfo.getTargetUsername());
- sourceDetails.put(DiskTO.CHAP_TARGET_SECRET, chapInfo.getTargetSecret());
+ volumeDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, chapInfo.getInitiatorUsername());
+ volumeDetails.put(DiskTO.CHAP_INITIATOR_SECRET, chapInfo.getInitiatorSecret());
+ volumeDetails.put(DiskTO.CHAP_TARGET_USERNAME, chapInfo.getTargetUsername());
+ volumeDetails.put(DiskTO.CHAP_TARGET_SECRET, chapInfo.getTargetSecret());
+ }
+
+ return volumeDetails;
+ }
+
+ private Map<String, String> getSnapshotDetails(SnapshotInfo snapshotInfo) {
+ Map<String, String> snapshotDetails = new HashMap<String, String>();
+
+ long storagePoolId = snapshotInfo.getDataStore().getId();
+ StoragePoolVO storagePoolVO = _storagePoolDao.findById(storagePoolId);
+
+ snapshotDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
+ snapshotDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
+
+ long snapshotId = snapshotInfo.getId();
+
+ snapshotDetails.put(DiskTO.IQN, getProperty(snapshotId, DiskTO.IQN));
+
+ snapshotDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, getProperty(snapshotId, DiskTO.CHAP_INITIATOR_USERNAME));
+ snapshotDetails.put(DiskTO.CHAP_INITIATOR_SECRET, getProperty(snapshotId, DiskTO.CHAP_INITIATOR_SECRET));
+ snapshotDetails.put(DiskTO.CHAP_TARGET_USERNAME, getProperty(snapshotId, DiskTO.CHAP_TARGET_USERNAME));
+ snapshotDetails.put(DiskTO.CHAP_TARGET_SECRET, getProperty(snapshotId, DiskTO.CHAP_TARGET_SECRET));
+
+ return snapshotDetails;
+ }
+
+ private HostVO getHost(SnapshotInfo snapshotInfo) {
+ HostVO hostVO = getHost(snapshotInfo.getDataCenterId(), true);
+
+ if (hostVO == null) {
+ hostVO = getHost(snapshotInfo.getDataCenterId(), false);
+
+ if (hostVO == null) {
+ throw new CloudRuntimeException("Unable to locate an applicable host");
+ }
}
- return sourceDetails;
+ return hostVO;
}
- public HostVO getHost(long dataStoreId) {
- StoragePoolVO storagePoolVO = _storagePoolDao.findById(dataStoreId);
+ private HostVO getHost(Long zoneId, boolean computeClusterMustSupportResign) {
+ if (zoneId == null) {
+ throw new CloudRuntimeException("Zone ID cannot be null.");
+ }
- List<? extends Cluster> clusters = _mgr.searchForClusters(storagePoolVO.getDataCenterId(), new Long(0), Long.MAX_VALUE, HypervisorType.XenServer.toString());
+ List<? extends Cluster> clusters = _mgr.searchForClusters(zoneId, new Long(0), Long.MAX_VALUE, HypervisorType.XenServer.toString());
if (clusters == null) {
- throw new CloudRuntimeException("Unable to locate an applicable cluster");
+ clusters = new ArrayList<>();
}
+ Collections.shuffle(clusters, new Random(System.nanoTime()));
+
+ clusters:
for (Cluster cluster : clusters) {
if (cluster.getAllocationState() == AllocationState.Enabled) {
List<HostVO> hosts = _hostDao.findByClusterId(cluster.getId());
if (hosts != null) {
+ Collections.shuffle(hosts, new Random(System.nanoTime()));
+
for (HostVO host : hosts) {
if (host.getResourceState() == ResourceState.Enabled) {
- return host;
+ if (computeClusterMustSupportResign) {
+ if (computeClusterSupportsResign(cluster.getId())) {
+ return host;
+ }
+ else {
+ // no other host in the cluster in question should be able to satisfy our requirements here, so move on to the next cluster
+ continue clusters;
+ }
+ }
+ else {
+ return host;
+ }
}
}
}
}
}
- throw new CloudRuntimeException("Unable to locate an applicable cluster");
+ return null;
}
@Override
public void copyAsync(Map<VolumeInfo, DataStore> volumeMap, VirtualMachineTO vmTo, Host srcHost, Host destHost, AsyncCompletionCallback<CopyCommandResult> callback) {
CopyCommandResult result = new CopyCommandResult(null, null);
+
result.setResult("Unsupported operation requested for copying data.");
+
callback.complete(result);
}
+
+ private Map<String, String> getDetails(DataObject dataObj) {
+ if (dataObj instanceof VolumeInfo) {
+ return getVolumeDetails((VolumeInfo)dataObj);
+ }
+ else if (dataObj instanceof SnapshotInfo) {
+ return getSnapshotDetails((SnapshotInfo)dataObj);
+ }
+
+ throw new CloudRuntimeException("'dataObj' must be of type 'VolumeInfo' or 'SnapshotInfo'.");
+ }
+
+ private CopyCmdAnswer performResignature(DataObject dataObj, HostVO hostVO) {
+ return performResignature(dataObj, hostVO, false);
+ }
+
+ private CopyCmdAnswer performResignature(DataObject dataObj, HostVO hostVO, boolean keepGrantedAccess) {
+ long storagePoolId = dataObj.getDataStore().getId();
+ DataStore dataStore = _dataStoreMgr.getDataStore(storagePoolId, DataStoreRole.Primary);
+
+ Map<String, String> details = getDetails(dataObj);
+
+ ResignatureCommand command = new ResignatureCommand(details);
+
+ ResignatureAnswer answer = null;
+
+ try {
+ _volumeService.grantAccess(dataObj, hostVO, dataStore);
+
+ answer = (ResignatureAnswer)_agentMgr.send(hostVO.getId(), command);
+ }
+ catch (Exception ex) {
+ keepGrantedAccess = false;
+
+ throw new CloudRuntimeException(ex.getMessage());
+ }
+ finally {
+ if (keepGrantedAccess == false) {
+ try {
+ _volumeService.revokeAccess(dataObj, hostVO, dataStore);
+ }
+ catch (Exception ex) {
+ s_logger.debug(ex.getMessage(), ex);
+ }
+ }
+ }
+
+ if (answer == null || !answer.getResult()) {
+ final String errMsg;
+
+ if (answer != null && answer.getDetails() != null && !answer.getDetails().isEmpty()) {
+ errMsg = answer.getDetails();
+ }
+ else {
+ errMsg = "Unable to perform resignature operation";
+ }
+
+ throw new CloudRuntimeException(errMsg);
+ }
+
+ VolumeObjectTO newVolume = new VolumeObjectTO();
+
+ newVolume.setSize(answer.getSize());
+ newVolume.setPath(answer.getPath());
+ newVolume.setFormat(answer.getFormat());
+
+ return new CopyCmdAnswer(newVolume);
+ }
+
+ private CopyCmdAnswer performCopyOfVdi(VolumeInfo volumeInfo, SnapshotInfo snapshotInfo, HostVO hostVO) {
+ String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
+ int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
+ CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), volumeInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+
+ CopyCmdAnswer copyCmdAnswer = null;
+
+ try {
+ _volumeService.grantAccess(snapshotInfo, hostVO, snapshotInfo.getDataStore());
+ _volumeService.grantAccess(volumeInfo, hostVO, volumeInfo.getDataStore());
+
+ Map<String, String> srcDetails = getSnapshotDetails(snapshotInfo);
+
+ copyCommand.setOptions(srcDetails);
+
+ Map<String, String> destDetails = getVolumeDetails(volumeInfo);
+
+ copyCommand.setOptions2(destDetails);
+
+ copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
+ }
+ catch (Exception ex) {
+ throw new CloudRuntimeException(ex.getMessage());
+ }
+ finally {
+ try {
+ _volumeService.revokeAccess(snapshotInfo, hostVO, snapshotInfo.getDataStore());
+ }
+ catch (Exception ex) {
+ s_logger.debug(ex.getMessage(), ex);
+ }
+
+ try {
+ _volumeService.revokeAccess(volumeInfo, hostVO, volumeInfo.getDataStore());
+ }
+ catch (Exception ex) {
+ s_logger.debug(ex.getMessage(), ex);
--- End diff --
Would an operator want to know if this rollback operation failed? It seems like it should be logged to ``WARN``. Also, please provide context information about the ``volumeInfo`` and ``host`` to assist with operational debugging.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by swill <gi...@git.apache.org>.
Github user swill commented on the pull request:
https://github.com/apache/cloudstack/pull/1403#issuecomment-210012980
Looking closer at this code. It look like `primaryStoreDriver` could easily be `null` in this case, so that return statement could potentially cause a null pointer exception...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by swill <gi...@git.apache.org>.
Github user swill commented on the pull request:
https://github.com/apache/cloudstack/pull/1403#issuecomment-207542455
I have not seen a style guide, but I have not looked. I have just seen a bunch of PRs recently and there seems to be a consistent work to remove the `_`s in the code, so I figured I would mention it. To me, this is not a reason not to merge the code, but I agree it would be good to know what the style should be so I can start enforcing it so we can start working towards consistent styling.
I am not requiring the LGTM code reviews to come from committers. If you are active on on Github and are adding value to the project by reviewing code, I am willing to work with that. Because I am not being strict about who is reviewing the code, I am also enforcing at least 2 people review the code so I get more eyes on ever PR.
As for the squash. It may make sense to have some history, but there are a lot of commits in there that are entirely useless outside the greater context of this PR. To me, the final version of the code merged trumps all. If it is squashed then we can more easily evaluate the changes associated with this commit later. If you think there are some points in the history of this feature that make sense to be called out as important, you can squash into 3 or 4 commits, but 43 commits for a single PR is crazy and will add more confusion than clarity.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by syed <gi...@git.apache.org>.
Github user syed commented on the pull request:
https://github.com/apache/cloudstack/pull/1403#issuecomment-184250180
LGTM. I've also been testing this in my dev environment with no issues so far
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by mike-tutkowski <gi...@git.apache.org>.
Github user mike-tutkowski commented on the pull request:
https://github.com/apache/cloudstack/pull/1403#issuecomment-180848498
Here's a copy of my Marvin integration tests (I added a .txt file type so that I could upload the file as it's not permitted to upload a file of type .py):
[TestSnapshots.py.txt](https://github.com/apache/cloudstack/files/120425/TestSnapshots.py.txt)
Here are the most recent test results:
[results.txt](https://github.com/apache/cloudstack/files/120426/results.txt)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r59082791
--- Diff: engine/storage/image/src/org/apache/cloudstack/storage/image/store/TemplateObject.java ---
@@ -189,12 +189,24 @@ public void processEvent(ObjectInDataStoreStateMachine.Event event, Answer answe
TemplateObjectTO newTemplate = (TemplateObjectTO)cpyAnswer.getNewData();
VMTemplateStoragePoolVO templatePoolRef = templatePoolDao.findByPoolTemplate(getDataStore().getId(), getId());
templatePoolRef.setDownloadPercent(100);
- if (newTemplate.getSize() != null) {
+
+ // In the case of managed storage, the template size may already be specified (by the storage plug-in), so do not overwrite it.
+ if (templatePoolRef.getTemplateSize() == 0 && newTemplate.getSize() != null) {
templatePoolRef.setTemplateSize(newTemplate.getSize());
}
+
templatePoolRef.setDownloadState(Status.DOWNLOADED);
- templatePoolRef.setLocalDownloadPath(newTemplate.getPath());
- templatePoolRef.setInstallPath(newTemplate.getPath());
+
+ // In the case of managed storage, the local download path may already be specified (by the storage plug-in), so do not overwrite it.
+ if (templatePoolRef.getLocalDownloadPath() == null) {
+ templatePoolRef.setLocalDownloadPath(newTemplate.getPath());
+ }
+
+ // In the case of managed storage, the install path may already be specified (by the storage plug-in), so do not overwrite it.
+ if (templatePoolRef.getInstallPath() == null) {
--- End diff --
Consider using ``Strings.isNullorEmpty`` to cover both the ``null`` and empty cases.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r59074950
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -172,78 +232,211 @@ private void validate(SnapshotInfo snapshotInfo) {
}
}
- private Void handleCreateTemplateFromSnapshot(SnapshotInfo snapshotInfo, TemplateInfo templateInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
+ private boolean usingBackendSnapshotFor(SnapshotInfo snapshotInfo) {
+ String property = getProperty(snapshotInfo.getId(), "takeSnapshot");
+
+ return Boolean.parseBoolean(property);
+ }
+
+ private void handleCreateTemplateFromSnapshot(SnapshotInfo snapshotInfo, TemplateInfo templateInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
try {
snapshotInfo.processEvent(Event.CopyingRequested);
}
catch (Exception ex) {
throw new CloudRuntimeException("This snapshot is not currently in a state where it can be used to create a template.");
}
- HostVO hostVO = getHost(snapshotInfo.getDataStore().getId());
- DataStore srcDataStore = snapshotInfo.getDataStore();
-
- String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
- int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
- CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), templateInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+ HostVO hostVO = getHost(snapshotInfo);
- String errMsg = null;
+ boolean usingBackendSnapshot = usingBackendSnapshotFor(snapshotInfo);
+ boolean computeClusterSupportsResign = computeClusterSupportsResign(hostVO.getClusterId());
- CopyCmdAnswer copyCmdAnswer = null;
+ if (usingBackendSnapshot && !computeClusterSupportsResign) {
+ throw new CloudRuntimeException("Unable to locate an applicable host with which to perform a resignature operation");
+ }
try {
- _volumeService.grantAccess(snapshotInfo, hostVO, srcDataStore);
+ if (usingBackendSnapshot) {
+ createVolumeFromSnapshot(hostVO, snapshotInfo, true);
+ }
- Map<String, String> srcDetails = getSnapshotDetails(_storagePoolDao.findById(srcDataStore.getId()), snapshotInfo);
+ DataStore srcDataStore = snapshotInfo.getDataStore();
- copyCommand.setOptions(srcDetails);
+ String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
+ int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
+ CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), templateInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+
+ String errMsg = null;
+
+ CopyCmdAnswer copyCmdAnswer = null;
- copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
- }
- catch (Exception ex) {
- throw new CloudRuntimeException(ex.getMessage());
- }
- finally {
try {
- _volumeService.revokeAccess(snapshotInfo, hostVO, srcDataStore);
+ // If we are using a back-end snapshot, then we should still have access to it from the hosts in the cluster that hostVO is in
+ // (because we passed in true as the third parameter to createVolumeFromSnapshot above).
+ if (usingBackendSnapshot == false) {
+ _volumeService.grantAccess(snapshotInfo, hostVO, srcDataStore);
+ }
+
+ Map<String, String> srcDetails = getSnapshotDetails(snapshotInfo);
+
+ copyCommand.setOptions(srcDetails);
+
+ copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
}
catch (Exception ex) {
--- End diff --
I know it's out of scope for this patch, but catching ``Exception`` is not a Good Thing(tm). Would it be possible to restrict the ``catch`` to only the expected checked exceptions?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by mike-tutkowski <gi...@git.apache.org>.
Github user mike-tutkowski commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r52114581
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -255,99 +332,149 @@ private Void handleCreateVolumeFromSnapshotBothOnStorageSystem(SnapshotInfo snap
VolumeApiResult result = future.get();
+ if (volumeDetail != null) {
+ _volumeDetailsDao.remove(volumeDetail.getId());
+ }
+
if (result.isFailed()) {
s_logger.debug("Failed to create a volume: " + result.getResult());
throw new CloudRuntimeException(result.getResult());
}
- }
- catch (Exception ex) {
- throw new CloudRuntimeException(ex.getMessage());
- }
-
- volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
- volumeInfo.processEvent(Event.MigrationRequested);
+ volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
- volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
+ volumeInfo.processEvent(Event.MigrationRequested);
- HostVO hostVO = getHost(snapshotInfo.getDataStore().getId());
+ volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
- String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
- int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
- CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), volumeInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+ if (useCloning) {
+ copyCmdAnswer = performResignature(volumeInfo, hostVO);
+ }
+ else {
+ // asking for a XenServer host here so we don't always prefer to use XenServer hosts that support resigning
+ // even when we don't need those hosts to do this kind of copy work
+ hostVO = getXenServerHost(snapshotInfo.getDataCenterId(), false);
- CopyCmdAnswer copyCmdAnswer = null;
+ copyCmdAnswer = performCopyOfVdi(volumeInfo, snapshotInfo, hostVO);
+ }
- try {
- _volumeService.grantAccess(snapshotInfo, hostVO, snapshotInfo.getDataStore());
- _volumeService.grantAccess(volumeInfo, hostVO, volumeInfo.getDataStore());
+ if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
+ if (copyCmdAnswer != null && copyCmdAnswer.getDetails() != null && !copyCmdAnswer.getDetails().isEmpty()) {
+ errMsg = copyCmdAnswer.getDetails();
+ }
+ else {
+ errMsg = "Unable to perform host-side operation";
+ }
+ }
+ }
+ catch (Exception ex) {
+ errMsg = ex.getMessage() != null ? ex.getMessage() : "Copy operation failed";
+ }
- Map<String, String> srcDetails = getSnapshotDetails(_storagePoolDao.findById(snapshotInfo.getDataStore().getId()), snapshotInfo);
+ CopyCommandResult result = new CopyCommandResult(null, copyCmdAnswer);
- copyCommand.setOptions(srcDetails);
+ result.setResult(errMsg);
- Map<String, String> destDetails = getVolumeDetails(volumeInfo);
+ callback.complete(result);
+ }
- copyCommand.setOptions2(destDetails);
+ // If the underlying storage system is making use of read-only snapshots, this gives the storage system the opportunity to
+ // create a volume from the snapshot so that we can copy the VHD file that should be inside of the snapshot to secondary storage.
+ //
+ // The resultant volume must be writable because we need to resign the SR and the VDI that should be inside of it before we copy
+ // the VHD file to secondary storage.
+ //
+ // If the storage system is using writable snapshots, then nothing need be done by that storage system here because we can just
+ // resign the SR and the VDI that should be inside of the snapshot before copying the VHD file to secondary storage.
+ private void createVolumeFromSnapshot(HostVO hostVO, SnapshotInfo snapshotInfo, boolean keepGrantedAccess) {
+ SnapshotDetailsVO snapshotDetails = handleSnapshotDetails(snapshotInfo.getId(), "tempVolume", "create");
- copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
- }
- catch (Exception ex) {
- throw new CloudRuntimeException(ex.getMessage());
+ try {
+ snapshotInfo.getDataStore().getDriver().createAsync(snapshotInfo.getDataStore(), snapshotInfo, null);
}
finally {
- try {
- _volumeService.revokeAccess(snapshotInfo, hostVO, snapshotInfo.getDataStore());
- }
- catch (Exception ex) {
- s_logger.debug(ex.getMessage(), ex);
- }
-
- try {
- _volumeService.revokeAccess(volumeInfo, hostVO, volumeInfo.getDataStore());
- }
- catch (Exception ex) {
- s_logger.debug(ex.getMessage(), ex);
- }
+ _snapshotDetailsDao.remove(snapshotDetails.getId());
}
- String errMsg = null;
+ CopyCmdAnswer copyCmdAnswer = performResignature(snapshotInfo, hostVO, keepGrantedAccess);
if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
if (copyCmdAnswer != null && copyCmdAnswer.getDetails() != null && !copyCmdAnswer.getDetails().isEmpty()) {
- errMsg = copyCmdAnswer.getDetails();
+ throw new CloudRuntimeException(copyCmdAnswer.getDetails());
}
else {
- errMsg = "Unable to perform host-side operation";
+ throw new CloudRuntimeException("Unable to perform host-side operation");
}
}
+ }
- CopyCommandResult result = new CopyCommandResult(null, copyCmdAnswer);
+ // If the underlying storage system needed to create a volume from a snapshot for createVolumeFromSnapshot(HostVO, SnapshotInfo), then
+ // this is its opportunity to delete that temporary volume and restore properties in snapshot_details to the way they were before the
+ // invocation of createVolumeFromSnapshot(HostVO, SnapshotInfo).
+ private void deleteVolumeFromSnapshot(SnapshotInfo snapshotInfo) {
--- End diff --
Done - thanks
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r59078742
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -361,59 +608,225 @@ private String getProperty(long snapshotId, String property) {
}
private Map<String, String> getVolumeDetails(VolumeInfo volumeInfo) {
- Map<String, String> sourceDetails = new HashMap<String, String>();
+ Map<String, String> volumeDetails = new HashMap<String, String>();
VolumeVO volumeVO = _volumeDao.findById(volumeInfo.getId());
long storagePoolId = volumeVO.getPoolId();
StoragePoolVO storagePoolVO = _storagePoolDao.findById(storagePoolId);
- sourceDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
- sourceDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
- sourceDetails.put(DiskTO.IQN, volumeVO.get_iScsiName());
+ volumeDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
+ volumeDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
+ volumeDetails.put(DiskTO.IQN, volumeVO.get_iScsiName());
ChapInfo chapInfo = _volumeService.getChapInfo(volumeInfo, volumeInfo.getDataStore());
if (chapInfo != null) {
- sourceDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, chapInfo.getInitiatorUsername());
- sourceDetails.put(DiskTO.CHAP_INITIATOR_SECRET, chapInfo.getInitiatorSecret());
- sourceDetails.put(DiskTO.CHAP_TARGET_USERNAME, chapInfo.getTargetUsername());
- sourceDetails.put(DiskTO.CHAP_TARGET_SECRET, chapInfo.getTargetSecret());
+ volumeDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, chapInfo.getInitiatorUsername());
+ volumeDetails.put(DiskTO.CHAP_INITIATOR_SECRET, chapInfo.getInitiatorSecret());
+ volumeDetails.put(DiskTO.CHAP_TARGET_USERNAME, chapInfo.getTargetUsername());
+ volumeDetails.put(DiskTO.CHAP_TARGET_SECRET, chapInfo.getTargetSecret());
+ }
+
+ return volumeDetails;
+ }
+
+ private Map<String, String> getSnapshotDetails(SnapshotInfo snapshotInfo) {
+ Map<String, String> snapshotDetails = new HashMap<String, String>();
+
+ long storagePoolId = snapshotInfo.getDataStore().getId();
+ StoragePoolVO storagePoolVO = _storagePoolDao.findById(storagePoolId);
+
+ snapshotDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
+ snapshotDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
+
+ long snapshotId = snapshotInfo.getId();
+
+ snapshotDetails.put(DiskTO.IQN, getProperty(snapshotId, DiskTO.IQN));
+
+ snapshotDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, getProperty(snapshotId, DiskTO.CHAP_INITIATOR_USERNAME));
+ snapshotDetails.put(DiskTO.CHAP_INITIATOR_SECRET, getProperty(snapshotId, DiskTO.CHAP_INITIATOR_SECRET));
+ snapshotDetails.put(DiskTO.CHAP_TARGET_USERNAME, getProperty(snapshotId, DiskTO.CHAP_TARGET_USERNAME));
+ snapshotDetails.put(DiskTO.CHAP_TARGET_SECRET, getProperty(snapshotId, DiskTO.CHAP_TARGET_SECRET));
+
+ return snapshotDetails;
+ }
+
+ private HostVO getHost(SnapshotInfo snapshotInfo) {
+ HostVO hostVO = getHost(snapshotInfo.getDataCenterId(), true);
+
+ if (hostVO == null) {
+ hostVO = getHost(snapshotInfo.getDataCenterId(), false);
+
+ if (hostVO == null) {
+ throw new CloudRuntimeException("Unable to locate an applicable host");
+ }
}
- return sourceDetails;
+ return hostVO;
}
- public HostVO getHost(long dataStoreId) {
- StoragePoolVO storagePoolVO = _storagePoolDao.findById(dataStoreId);
+ private HostVO getHost(Long zoneId, boolean computeClusterMustSupportResign) {
+ if (zoneId == null) {
+ throw new CloudRuntimeException("Zone ID cannot be null.");
+ }
- List<? extends Cluster> clusters = _mgr.searchForClusters(storagePoolVO.getDataCenterId(), new Long(0), Long.MAX_VALUE, HypervisorType.XenServer.toString());
+ List<? extends Cluster> clusters = _mgr.searchForClusters(zoneId, new Long(0), Long.MAX_VALUE, HypervisorType.XenServer.toString());
--- End diff --
``new Long(0)`` avoids the constant pool -- placing unnecessary pressure on the garbage collector. With autoboxing, ``0L`` will accomplish the same thing without skipping the constant pool.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by DaanHoogland <gi...@git.apache.org>.
Github user DaanHoogland commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r62892495
--- Diff: server/src/com/cloud/storage/snapshot/SnapshotManagerImpl.java ---
@@ -144,6 +145,8 @@
@Inject
SnapshotDataStoreDao _snapshotStoreDao;
@Inject
+ private DataStoreManager _dataStoreMgr;
--- End diff --
there is already a dataStoreMgr. why are two needed? (also underscore names are against naming convention)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on the pull request:
https://github.com/apache/cloudstack/pull/1403#issuecomment-207565521
@mike-tutkowski while we can't run the tests, there is still value in being able to read them to understand how the functionality is being tested. During review, we may identify scenarios that have not been covered that would be valuable to add. Therefore, I think there is value in having the test code even if it cannot be run.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by mike-tutkowski <gi...@git.apache.org>.
Github user mike-tutkowski commented on the pull request:
https://github.com/apache/cloudstack/pull/1403#issuecomment-216354858
After rebasing today, I re-ran SolidFire-related automated integration tests (about 3.5 hours of tests) and all were successful.
test_01_create_volume_snapshot_using_sf_snapshot (TestSnapshots.TestSnapshots) ... === TestName: test_01_create_volume_snapshot_using_sf_snapshot | Status : SUCCESS ===
ok
test_02_create_volume_snapshot_using_sf_volume (TestSnapshots.TestSnapshots) ... === TestName: test_02_create_volume_snapshot_using_sf_volume | Status : SUCCESS ===
ok
test_03_create_volume_snapshot_using_sf_volume_and_sf_snapshot (TestSnapshots.TestSnapshots) ... === TestName: test_03_create_volume_snapshot_using_sf_volume_and_sf_snapshot | Status : SUCCESS ===
ok
----------------------------------------------------------------------
Ran 3 tests in 7409.770s
OK
test_00_check_template_cache (TestVolumes.TestVolumes) ... === TestName: test_00_check_template_cache | Status : SUCCESS ===
ok
Attach a volume to a stopped virtual machine, then start VM ... === TestName: test_01_attach_new_volume_to_stopped_VM | Status : SUCCESS ===
ok
Attach, detach, and attach volume to a running VM ... === TestName: test_02_attach_detach_attach_volume | Status : SUCCESS ===
ok
Attach volume to running VM, then reboot. ... === TestName: test_03_attached_volume_reboot_VM | Status : SUCCESS ===
ok
Detach volume from a running VM, then reboot. ... === TestName: test_04_detach_volume_reboot | Status : SUCCESS ===
ok
Detach volume from a stopped VM, then start. ... === TestName: test_05_detach_vol_stopped_VM_start | Status : SUCCESS ===
ok
Attach a volume to a stopped virtual machine, then start VM ... === TestName: test_06_attach_volume_to_stopped_VM | Status : SUCCESS ===
ok
Destroy and expunge VM with attached volume ... === TestName: test_07_destroy_expunge_VM_with_volume | Status : SUCCESS ===
ok
Delete volume that was attached to a VM and is detached now ... === TestName: test_08_delete_volume_was_attached | Status : SUCCESS ===
ok
Attach a data disk to a VM in one account and attach another data disk to a VM in another account ... === TestName: test_09_attach_volumes_multiple_accounts | Status : SUCCESS ===
ok
Attach more than one disk to a VM ... === TestName: test_10_attach_more_than_one_disk_to_VM | Status : SUCCESS ===
ok
----------------------------------------------------------------------
Ran 11 tests in 1982.066s
OK
test_00_check_template_cache (TestVolumes.TestVolumes) ... === TestName: test_00_check_template_cache | Status : SUCCESS ===
ok
Attach a volume to a stopped virtual machine, then start VM ... === TestName: test_01_attach_new_volume_to_stopped_VM | Status : SUCCESS ===
ok
Attach, detach, and attach volume to a running VM ... === TestName: test_02_attach_detach_attach_volume | Status : SUCCESS ===
ok
Attach volume to running VM, then reboot. ... === TestName: test_03_attached_volume_reboot_VM | Status : SUCCESS ===
ok
Detach volume from a running VM, then reboot. ... === TestName: test_04_detach_volume_reboot | Status : SUCCESS ===
ok
Detach volume from a stopped VM, then start. ... === TestName: test_05_detach_vol_stopped_VM_start | Status : SUCCESS ===
ok
Attach a volume to a stopped virtual machine, then start VM ... === TestName: test_06_attach_volume_to_stopped_VM | Status : SUCCESS ===
ok
Destroy and expunge VM with attached volume ... === TestName: test_07_destroy_expunge_VM_with_volume | Status : SUCCESS ===
ok
Delete volume that was attached to a VM and is detached now ... === TestName: test_08_delete_volume_was_attached | Status : SUCCESS ===
ok
Attach a data disk to a VM in one account and attach another data disk to a VM in another account ... === TestName: test_09_attach_volumes_multiple_accounts | Status : SUCCESS ===
ok
Attach more than one disk to a VM ... === TestName: test_10_attach_more_than_one_disk_to_VM | Status : SUCCESS ===
ok
----------------------------------------------------------------------
Ran 11 tests in 2352.461s
OK
test_01_take_VM_snapshot (TestVMSnapshots.TestVMSnapshots) ... === TestName: test_01_take_VM_snapshot | Status : SUCCESS ===
ok
test_02_take_VM_snapshot_with_data_disk (TestVMSnapshots.TestVMSnapshots) ... === TestName: test_02_take_VM_snapshot_with_data_disk | Status : SUCCESS ===
ok
----------------------------------------------------------------------
Ran 2 tests in 789.729s
OK
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r59084080
--- Diff: engine/storage/snapshot/src/org/apache/cloudstack/storage/snapshot/StorageSystemSnapshotStrategy.java ---
@@ -245,6 +228,52 @@ public SnapshotInfo takeSnapshot(SnapshotInfo snapshotInfo) {
return snapshotInfo;
}
+ private boolean canStorageSystemCreateVolumeFromSnapshot(long storagePoolId) {
+ boolean supportsCloningVolumeFromSnapshot = false;
+
+ DataStore dataStore = _dataStoreMgr.getDataStore(storagePoolId, DataStoreRole.Primary);
+
+ Map<String, String> mapCapabilities = dataStore.getDriver().getCapabilities();
+
+ if (mapCapabilities != null) {
+ String value = mapCapabilities.get(DataStoreCapabilities.CAN_CREATE_VOLUME_FROM_SNAPSHOT.toString());
+
+ supportsCloningVolumeFromSnapshot = new Boolean(value);
+ }
+
+ return supportsCloningVolumeFromSnapshot;
+ }
+
+ private boolean computeClusterSupportsResign(long clusterId) {
+ List<HostVO> hosts = _hostDao.findByClusterId(clusterId);
+
+ if (hosts == null) {
+ return false;
+ }
+
+ for (HostVO host : hosts) {
+ if (host == null) {
+ return false;
+ }
+
+ DetailVO hostDetail = _hostDetailsDao.findDetail(host.getId(), "supportsResign");
+
+ if (hostDetail == null) {
+ return false;
+ }
+
+ String value = hostDetail.getValue();
+
+ Boolean booleanValue = Boolean.valueOf(value);
+
+ if (booleanValue == false) {
+ return false;
+ }
+ }
+
+ return true;
+ }
+
--- End diff --
This method seems to a duplicate of a method on another class. Can the implementations be consolidated for consistency and better maintenance?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by rafaelweingartner <gi...@git.apache.org>.
Github user rafaelweingartner commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r59077175
--- Diff: engine/api/src/org/apache/cloudstack/engine/subsystem/api/storage/PrimaryDataStoreDriver.java ---
@@ -23,18 +23,21 @@
import com.cloud.host.Host;
import com.cloud.storage.StoragePool;
-import com.cloud.storage.Volume;
public interface PrimaryDataStoreDriver extends DataStoreDriver {
- public ChapInfo getChapInfo(VolumeInfo volumeInfo);
+ public ChapInfo getChapInfo(DataObject dataObject);
public boolean grantAccess(DataObject dataObject, Host host, DataStore dataStore);
public void revokeAccess(DataObject dataObject, Host host, DataStore dataStore);
// intended for managed storage (cloud.storage_pool.managed = true)
// if not managed, return volume.getSize()
- public long getVolumeSizeIncludingHypervisorSnapshotReserve(Volume volume, StoragePool storagePool);
+ public long getDataObjectSizeIncludingHypervisorSnapshotReserve(DataObject dataObject, StoragePool storagePool);
+
+ // intended for zone-wide primary storage that is capable of storing a template once and using it in multiple clusters
+ // if not this kind of storage, return 0
+ public long getBytesRequiredForTemplate(TemplateInfo templateInfo, StoragePool storagePool);
--- End diff --
This is not a proper documentation.
I would suggest changing from Java comments style to Javadoc.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r59078437
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -361,59 +608,225 @@ private String getProperty(long snapshotId, String property) {
}
private Map<String, String> getVolumeDetails(VolumeInfo volumeInfo) {
- Map<String, String> sourceDetails = new HashMap<String, String>();
+ Map<String, String> volumeDetails = new HashMap<String, String>();
VolumeVO volumeVO = _volumeDao.findById(volumeInfo.getId());
long storagePoolId = volumeVO.getPoolId();
StoragePoolVO storagePoolVO = _storagePoolDao.findById(storagePoolId);
- sourceDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
- sourceDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
- sourceDetails.put(DiskTO.IQN, volumeVO.get_iScsiName());
+ volumeDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
+ volumeDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
+ volumeDetails.put(DiskTO.IQN, volumeVO.get_iScsiName());
ChapInfo chapInfo = _volumeService.getChapInfo(volumeInfo, volumeInfo.getDataStore());
if (chapInfo != null) {
- sourceDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, chapInfo.getInitiatorUsername());
- sourceDetails.put(DiskTO.CHAP_INITIATOR_SECRET, chapInfo.getInitiatorSecret());
- sourceDetails.put(DiskTO.CHAP_TARGET_USERNAME, chapInfo.getTargetUsername());
- sourceDetails.put(DiskTO.CHAP_TARGET_SECRET, chapInfo.getTargetSecret());
+ volumeDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, chapInfo.getInitiatorUsername());
+ volumeDetails.put(DiskTO.CHAP_INITIATOR_SECRET, chapInfo.getInitiatorSecret());
+ volumeDetails.put(DiskTO.CHAP_TARGET_USERNAME, chapInfo.getTargetUsername());
+ volumeDetails.put(DiskTO.CHAP_TARGET_SECRET, chapInfo.getTargetSecret());
+ }
+
+ return volumeDetails;
+ }
+
+ private Map<String, String> getSnapshotDetails(SnapshotInfo snapshotInfo) {
+ Map<String, String> snapshotDetails = new HashMap<String, String>();
+
+ long storagePoolId = snapshotInfo.getDataStore().getId();
+ StoragePoolVO storagePoolVO = _storagePoolDao.findById(storagePoolId);
+
+ snapshotDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
+ snapshotDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
+
+ long snapshotId = snapshotInfo.getId();
+
+ snapshotDetails.put(DiskTO.IQN, getProperty(snapshotId, DiskTO.IQN));
+
+ snapshotDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, getProperty(snapshotId, DiskTO.CHAP_INITIATOR_USERNAME));
+ snapshotDetails.put(DiskTO.CHAP_INITIATOR_SECRET, getProperty(snapshotId, DiskTO.CHAP_INITIATOR_SECRET));
+ snapshotDetails.put(DiskTO.CHAP_TARGET_USERNAME, getProperty(snapshotId, DiskTO.CHAP_TARGET_USERNAME));
+ snapshotDetails.put(DiskTO.CHAP_TARGET_SECRET, getProperty(snapshotId, DiskTO.CHAP_TARGET_SECRET));
+
+ return snapshotDetails;
+ }
+
+ private HostVO getHost(SnapshotInfo snapshotInfo) {
+ HostVO hostVO = getHost(snapshotInfo.getDataCenterId(), true);
+
+ if (hostVO == null) {
+ hostVO = getHost(snapshotInfo.getDataCenterId(), false);
+
+ if (hostVO == null) {
+ throw new CloudRuntimeException("Unable to locate an applicable host");
--- End diff --
Please include the datacenter id in the message to assist operational debugging.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by mike-tutkowski <gi...@git.apache.org>.
Github user mike-tutkowski commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r63131338
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -255,99 +448,153 @@ private Void handleCreateVolumeFromSnapshotBothOnStorageSystem(SnapshotInfo snap
VolumeApiResult result = future.get();
+ if (volumeDetail != null) {
+ _volumeDetailsDao.remove(volumeDetail.getId());
+ }
+
if (result.isFailed()) {
s_logger.debug("Failed to create a volume: " + result.getResult());
throw new CloudRuntimeException(result.getResult());
}
- }
- catch (Exception ex) {
- throw new CloudRuntimeException(ex.getMessage());
- }
- volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
+ volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
- volumeInfo.processEvent(Event.MigrationRequested);
+ volumeInfo.processEvent(Event.MigrationRequested);
- volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
+ volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
- HostVO hostVO = getHost(snapshotInfo.getDataStore().getId());
+ if (useCloning) {
+ copyCmdAnswer = performResignature(volumeInfo, hostVO);
+ }
+ else {
+ // asking for a XenServer host here so we don't always prefer to use XenServer hosts that support resigning
+ // even when we don't need those hosts to do this kind of copy work
+ hostVO = getHost(snapshotInfo.getDataCenterId(), false);
- String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
- int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
- CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), volumeInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+ copyCmdAnswer = performCopyOfVdi(volumeInfo, snapshotInfo, hostVO);
+ }
- CopyCmdAnswer copyCmdAnswer = null;
+ if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
+ if (copyCmdAnswer != null && !StringUtils.isEmpty(copyCmdAnswer.getDetails())) {
+ errMsg = copyCmdAnswer.getDetails();
+ }
+ else {
+ errMsg = "Unable to perform host-side operation";
+ }
+ }
+ }
+ catch (Exception ex) {
+ errMsg = ex.getMessage() != null ? ex.getMessage() : "Copy operation failed";
--- End diff --
Done
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r62892193
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -172,78 +232,211 @@ private void validate(SnapshotInfo snapshotInfo) {
}
}
- private Void handleCreateTemplateFromSnapshot(SnapshotInfo snapshotInfo, TemplateInfo templateInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
+ private boolean usingBackendSnapshotFor(SnapshotInfo snapshotInfo) {
+ String property = getProperty(snapshotInfo.getId(), "takeSnapshot");
+
+ return Boolean.parseBoolean(property);
+ }
+
+ private void handleCreateTemplateFromSnapshot(SnapshotInfo snapshotInfo, TemplateInfo templateInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
try {
snapshotInfo.processEvent(Event.CopyingRequested);
}
catch (Exception ex) {
throw new CloudRuntimeException("This snapshot is not currently in a state where it can be used to create a template.");
}
- HostVO hostVO = getHost(snapshotInfo.getDataStore().getId());
- DataStore srcDataStore = snapshotInfo.getDataStore();
-
- String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
- int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
- CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), templateInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+ HostVO hostVO = getHost(snapshotInfo);
- String errMsg = null;
+ boolean usingBackendSnapshot = usingBackendSnapshotFor(snapshotInfo);
+ boolean computeClusterSupportsResign = computeClusterSupportsResign(hostVO.getClusterId());
- CopyCmdAnswer copyCmdAnswer = null;
+ if (usingBackendSnapshot && !computeClusterSupportsResign) {
+ throw new CloudRuntimeException("Unable to locate an applicable host with which to perform a resignature operation");
+ }
try {
- _volumeService.grantAccess(snapshotInfo, hostVO, srcDataStore);
+ if (usingBackendSnapshot) {
+ createVolumeFromSnapshot(hostVO, snapshotInfo, true);
+ }
- Map<String, String> srcDetails = getSnapshotDetails(_storagePoolDao.findById(srcDataStore.getId()), snapshotInfo);
+ DataStore srcDataStore = snapshotInfo.getDataStore();
- copyCommand.setOptions(srcDetails);
+ String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
+ int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
+ CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), templateInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+
+ String errMsg = null;
+
+ CopyCmdAnswer copyCmdAnswer = null;
- copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
- }
- catch (Exception ex) {
- throw new CloudRuntimeException(ex.getMessage());
- }
- finally {
try {
- _volumeService.revokeAccess(snapshotInfo, hostVO, srcDataStore);
+ // If we are using a back-end snapshot, then we should still have access to it from the hosts in the cluster that hostVO is in
+ // (because we passed in true as the third parameter to createVolumeFromSnapshot above).
+ if (usingBackendSnapshot == false) {
+ _volumeService.grantAccess(snapshotInfo, hostVO, srcDataStore);
+ }
+
+ Map<String, String> srcDetails = getSnapshotDetails(snapshotInfo);
+
+ copyCommand.setOptions(srcDetails);
+
+ copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
}
catch (Exception ex) {
- s_logger.debug(ex.getMessage(), ex);
+ throw new CloudRuntimeException(ex.getMessage());
}
+ finally {
+ try {
+ _volumeService.revokeAccess(snapshotInfo, hostVO, srcDataStore);
+ }
+ catch (Exception ex) {
+ s_logger.debug(ex.getMessage(), ex);
+ }
- if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
- if (copyCmdAnswer != null && copyCmdAnswer.getDetails() != null && !copyCmdAnswer.getDetails().isEmpty()) {
- errMsg = copyCmdAnswer.getDetails();
+ if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
+ if (copyCmdAnswer != null && !StringUtils.isEmpty(copyCmdAnswer.getDetails())) {
+ errMsg = copyCmdAnswer.getDetails();
+ }
+ else {
+ errMsg = "Unable to perform host-side operation";
+ }
}
- else {
- errMsg = "Unable to perform host-side operation";
+
+ try {
+ if (errMsg == null) {
+ snapshotInfo.processEvent(Event.OperationSuccessed);
+ }
+ else {
+ snapshotInfo.processEvent(Event.OperationFailed);
+ }
+ }
+ catch (Exception ex) {
+ s_logger.debug(ex.getMessage(), ex);
}
}
- try {
- if (errMsg == null) {
- snapshotInfo.processEvent(Event.OperationSuccessed);
+ CopyCommandResult result = new CopyCommandResult(null, copyCmdAnswer);
+
+ result.setResult(errMsg);
+
+ callback.complete(result);
+ }
+ finally {
+ if (usingBackendSnapshot) {
+ deleteVolumeFromSnapshot(snapshotInfo);
+ }
+ }
+ }
+
+ /**
+ * Clones a template present on the storage to a new volume and resignatures it.
+ *
+ * @param srcData source template
+ * @param destData destination ROOT volume
+ * @param callback for async
+ */
+ private void handleCreateVolumeFromTemplateBothOnStorageSystem(DataObject srcData, DataObject destData, AsyncCompletionCallback<CopyCommandResult> callback) {
+ CopyCmdAnswer copyCmdAnswer = null;
+ String errMsg = null;
+
+ TemplateInfo templateInfo = (TemplateInfo)srcData;
+ VolumeInfo volumeInfo = (VolumeInfo)destData;
--- End diff --
@DaanHoogland the ``checkArgument`` calls protects against an NPE, but do not check the types to avoid a ``ClassCastException``. Adding ``Preconditions.checkArgument(srcData instanceof TemplateInfo, "<useful error message>")`` and ``Preconditions.checkArgument(destData instanceof VolumeInfo, "<useful error message>")``.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by bvbharatk <gi...@git.apache.org>.
Github user bvbharatk commented on the pull request:
https://github.com/apache/cloudstack/pull/1403#issuecomment-196016247
### ACS CI BVT Run
**Sumarry:**
Build Number 90
Hypervisor xenserver
NetworkType Advanced
Passed=99
Failed=20
Skipped=4
**The follwing tests have known issues**
integration.smoke.test_iso.TestISO.test_07_list_default_iso
integration.smoke.test_privategw_acl.py
integration.smoke.test_vm_snapshots.TestSnapshots.test_01_test_vm_volume_snapshot
integration.smoke.test_vpc_vpn.TestVpcSite2SiteVpn.test_vpc_site2site_vpn
integration.smoke.test_iso.TestISO.test_04_extract_Iso
_Link to logs Folder (search by build_no):_ https://www.dropbox.com/sh/yj3wnzbceo9uef2/AAB6u-Iap-xztdm6jHX9SjPja?dl=0
**Failed tests:**
* integration.smoke.test_vpc_vpn.TestVpcRemoteAccessVpn
* test_vpc_remote_access_vpn
* test_vpc_site2site_vpn
* integration.smoke.test_vm_snapshots.TestSnapshots
* test_01_test_vm_volume_snapshot
* integration.smoke.test_privategw_acl.TestPrivateGwACL
* test_02_vpc_privategw_static_routes
* test_03_rvpc_privategw_static_routes
* integration.smoke.test_guest_vlan_range.TestDedicateGuestVlanRange
* test_dedicateGuestVlanRange
* ContextSuite context=TestDedicateGuestVlanRange>:teardown
* integration.smoke.test_loadbalance.TestLoadBalance
* test_01_create_lb_rule_src_nat
* test_02_create_lb_rule_non_nat
* test_assign_and_removal_lb
* <nose.suite
* ContextSuite context=TestNiciraContoller>:setup
* integration.smoke.test_primary_storage.TestPrimaryStorageServices
* test_01_primary_storage_iscsi
* integration.smoke.test_network.TestDeleteAccount
* test_delete_account
* integration.smoke.test_non_contigiousvlan.TestUpdatePhysicalNetwork
* test_extendPhysicalNetworkVlan
* integration.smoke.test_affinity_groups.TestDeployVmWithAffinityGroup
* test_DeployVmAntiAffinityGroup
* integration.smoke.test_internal_lb.TestInternalLb
* test02_internallb_haproxy_stats_on_all_interfaces
* test_01_internallb_roundrobin_1VPC_3VM_HTTP_port80
* integration.smoke.test_templates.TestCreateTemplate
* test_04_extract_template
* integration.smoke.test_iso.TestCreateIso
* test_04_extract_Iso
* test_07_list_default_iso
**Skipped tests:**
test_vm_nic_adapter_vmxnet3
test_deploy_vgpu_enabled_vm
test_06_copy_template
test_06_copy_iso
**Passed test suits:**
integration.smoke.test_deploy_vm_with_userdata.TestDeployVmWithUserData
integration.smoke.test_affinity_groups_projects.TestDeployVmWithAffinityGroup
integration.smoke.test_portable_publicip.TestPortablePublicIPAcquire
integration.smoke.test_over_provisioning.TestUpdateOverProvision
integration.smoke.test_global_settings.TestUpdateConfigWithScope
integration.smoke.test_scale_vm.TestScaleVm
integration.smoke.test_service_offerings.TestCreateServiceOffering
integration.smoke.test_routers.TestRouterServices
integration.smoke.test_reset_vm_on_reboot.TestResetVmOnReboot
integration.smoke.test_snapshots.TestSnapshotRootDisk
integration.smoke.test_deploy_vms_with_varied_deploymentplanners.TestDeployVmWithVariedPlanners
integration.smoke.test_deploy_vm_iso.TestDeployVMFromISO
integration.smoke.test_public_ip_range.TestDedicatePublicIPRange
integration.smoke.test_multipleips_per_nic.TestDeployVM
integration.smoke.test_regions.TestRegions
integration.smoke.test_network_acl.TestNetworkACL
integration.smoke.test_pvlan.TestPVLAN
integration.smoke.test_volumes.TestCreateVolume
integration.smoke.test_ssvm.TestSSVMs
integration.smoke.test_nic.TestNic
integration.smoke.test_deploy_vm_root_resize.TestDeployVM
integration.smoke.test_resource_detail.TestResourceDetail
integration.smoke.test_secondary_storage.TestSecStorageServices
integration.smoke.test_vm_life_cycle.TestDeployVM
integration.smoke.test_disk_offerings.TestCreateDiskOffering
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by mike-tutkowski <gi...@git.apache.org>.
Github user mike-tutkowski commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r63082964
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -255,99 +473,123 @@ private Void handleCreateVolumeFromSnapshotBothOnStorageSystem(SnapshotInfo snap
VolumeApiResult result = future.get();
+ if (volumeDetail != null) {
+ _volumeDetailsDao.remove(volumeDetail.getId());
+ }
+
if (result.isFailed()) {
- s_logger.debug("Failed to create a volume: " + result.getResult());
+ s_logger.warn("Failed to create a volume: " + result.getResult());
throw new CloudRuntimeException(result.getResult());
}
- }
- catch (Exception ex) {
- throw new CloudRuntimeException(ex.getMessage());
- }
- volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
+ volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
- volumeInfo.processEvent(Event.MigrationRequested);
+ volumeInfo.processEvent(Event.MigrationRequested);
- volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
+ volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
- HostVO hostVO = getHost(snapshotInfo.getDataStore().getId());
-
- String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
- int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
- CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), volumeInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+ if (useCloning) {
+ copyCmdAnswer = performResignature(volumeInfo, hostVO);
+ }
+ else {
+ // asking for a XenServer host here so we don't always prefer to use XenServer hosts that support resigning
+ // even when we don't need those hosts to do this kind of copy work
+ hostVO = getHost(snapshotInfo.getDataCenterId(), false);
- CopyCmdAnswer copyCmdAnswer = null;
+ copyCmdAnswer = performCopyOfVdi(volumeInfo, snapshotInfo, hostVO);
+ }
- try {
- _volumeService.grantAccess(snapshotInfo, hostVO, snapshotInfo.getDataStore());
- _volumeService.grantAccess(volumeInfo, hostVO, volumeInfo.getDataStore());
+ if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
+ if (copyCmdAnswer != null && !StringUtils.isEmpty(copyCmdAnswer.getDetails())) {
+ errMsg = copyCmdAnswer.getDetails();
+ }
+ else {
+ errMsg = "Unable to create volume from snapshot";
+ }
+ }
+ }
+ catch (Exception ex) {
+ errMsg = ex.getMessage() != null ? ex.getMessage() : "Copy operation failed";
+ }
- Map<String, String> srcDetails = getSnapshotDetails(_storagePoolDao.findById(snapshotInfo.getDataStore().getId()), snapshotInfo);
+ CopyCommandResult result = new CopyCommandResult(null, copyCmdAnswer);
- copyCommand.setOptions(srcDetails);
+ result.setResult(errMsg);
- Map<String, String> destDetails = getVolumeDetails(volumeInfo);
+ callback.complete(result);
+ }
- copyCommand.setOptions2(destDetails);
+ /**
+ * If the underlying storage system is making use of read-only snapshots, this gives the storage system the opportunity to
+ * create a volume from the snapshot so that we can copy the VHD file that should be inside of the snapshot to secondary storage.
+ *
+ * The resultant volume must be writable because we need to resign the SR and the VDI that should be inside of it before we copy
+ * the VHD file to secondary storage.
+ *
+ * If the storage system is using writable snapshots, then nothing need be done by that storage system here because we can just
+ * resign the SR and the VDI that should be inside of the snapshot before copying the VHD file to secondary storage.
+ */
+ private void createVolumeFromSnapshot(HostVO hostVO, SnapshotInfo snapshotInfo, boolean keepGrantedAccess) {
+ SnapshotDetailsVO snapshotDetails = handleSnapshotDetails(snapshotInfo.getId(), "tempVolume", "create");
- copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
- }
- catch (Exception ex) {
- throw new CloudRuntimeException(ex.getMessage());
+ try {
+ snapshotInfo.getDataStore().getDriver().createAsync(snapshotInfo.getDataStore(), snapshotInfo, null);
}
finally {
- try {
- _volumeService.revokeAccess(snapshotInfo, hostVO, snapshotInfo.getDataStore());
- }
- catch (Exception ex) {
- s_logger.debug(ex.getMessage(), ex);
- }
-
- try {
- _volumeService.revokeAccess(volumeInfo, hostVO, volumeInfo.getDataStore());
- }
- catch (Exception ex) {
- s_logger.debug(ex.getMessage(), ex);
- }
+ _snapshotDetailsDao.remove(snapshotDetails.getId());
}
- String errMsg = null;
+ CopyCmdAnswer copyCmdAnswer = performResignature(snapshotInfo, hostVO, keepGrantedAccess);
if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
- if (copyCmdAnswer != null && copyCmdAnswer.getDetails() != null && !copyCmdAnswer.getDetails().isEmpty()) {
- errMsg = copyCmdAnswer.getDetails();
+ if (copyCmdAnswer != null && !StringUtils.isEmpty(copyCmdAnswer.getDetails())) {
+ throw new CloudRuntimeException(copyCmdAnswer.getDetails());
}
else {
- errMsg = "Unable to perform host-side operation";
+ throw new CloudRuntimeException("Unable to create volume from snapshot");
}
}
+ }
- CopyCommandResult result = new CopyCommandResult(null, copyCmdAnswer);
+ /**
+ * If the underlying storage system needed to create a volume from a snapshot for createVolumeFromSnapshot(HostVO, SnapshotInfo), then
+ * this is its opportunity to delete that temporary volume and restore properties in snapshot_details to the way they were before the
--- End diff --
@DaanHoogland I converted that comment to JavaDoc due to another reviewer's comment. :) If you'd like, I can put it back the way I initially had it.
I don't quite understand your first sentence here, though.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by mike-tutkowski <gi...@git.apache.org>.
Github user mike-tutkowski commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r63132551
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -361,59 +600,205 @@ private String getProperty(long snapshotId, String property) {
}
private Map<String, String> getVolumeDetails(VolumeInfo volumeInfo) {
- Map<String, String> sourceDetails = new HashMap<String, String>();
+ Map<String, String> volumeDetails = new HashMap<String, String>();
VolumeVO volumeVO = _volumeDao.findById(volumeInfo.getId());
long storagePoolId = volumeVO.getPoolId();
StoragePoolVO storagePoolVO = _storagePoolDao.findById(storagePoolId);
- sourceDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
- sourceDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
- sourceDetails.put(DiskTO.IQN, volumeVO.get_iScsiName());
+ volumeDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
+ volumeDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
+ volumeDetails.put(DiskTO.IQN, volumeVO.get_iScsiName());
ChapInfo chapInfo = _volumeService.getChapInfo(volumeInfo, volumeInfo.getDataStore());
if (chapInfo != null) {
- sourceDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, chapInfo.getInitiatorUsername());
- sourceDetails.put(DiskTO.CHAP_INITIATOR_SECRET, chapInfo.getInitiatorSecret());
- sourceDetails.put(DiskTO.CHAP_TARGET_USERNAME, chapInfo.getTargetUsername());
- sourceDetails.put(DiskTO.CHAP_TARGET_SECRET, chapInfo.getTargetSecret());
+ volumeDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, chapInfo.getInitiatorUsername());
+ volumeDetails.put(DiskTO.CHAP_INITIATOR_SECRET, chapInfo.getInitiatorSecret());
+ volumeDetails.put(DiskTO.CHAP_TARGET_USERNAME, chapInfo.getTargetUsername());
+ volumeDetails.put(DiskTO.CHAP_TARGET_SECRET, chapInfo.getTargetSecret());
}
- return sourceDetails;
+ return volumeDetails;
}
- public HostVO getHost(long dataStoreId) {
- StoragePoolVO storagePoolVO = _storagePoolDao.findById(dataStoreId);
+ private Map<String, String> getSnapshotDetails(SnapshotInfo snapshotInfo) {
+ Map<String, String> snapshotDetails = new HashMap<String, String>();
+
+ long storagePoolId = snapshotInfo.getDataStore().getId();
+ StoragePoolVO storagePoolVO = _storagePoolDao.findById(storagePoolId);
+
+ snapshotDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
+ snapshotDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
+
+ long snapshotId = snapshotInfo.getId();
+
+ snapshotDetails.put(DiskTO.IQN, getProperty(snapshotId, DiskTO.IQN));
- List<? extends Cluster> clusters = _mgr.searchForClusters(storagePoolVO.getDataCenterId(), new Long(0), Long.MAX_VALUE, HypervisorType.XenServer.toString());
+ snapshotDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, getProperty(snapshotId, DiskTO.CHAP_INITIATOR_USERNAME));
+ snapshotDetails.put(DiskTO.CHAP_INITIATOR_SECRET, getProperty(snapshotId, DiskTO.CHAP_INITIATOR_SECRET));
+ snapshotDetails.put(DiskTO.CHAP_TARGET_USERNAME, getProperty(snapshotId, DiskTO.CHAP_TARGET_USERNAME));
+ snapshotDetails.put(DiskTO.CHAP_TARGET_SECRET, getProperty(snapshotId, DiskTO.CHAP_TARGET_SECRET));
- if (clusters == null) {
- throw new CloudRuntimeException("Unable to locate an applicable cluster");
+ return snapshotDetails;
+ }
+
+ private HostVO getHost(SnapshotInfo snapshotInfo) {
+ HostVO hostVO = getHost(snapshotInfo.getDataCenterId(), true);
+
+ if (hostVO == null) {
+ hostVO = getHost(snapshotInfo.getDataCenterId(), false);
+
+ if (hostVO == null) {
+ throw new CloudRuntimeException("Unable to locate an applicable host in data center with ID = " + snapshotInfo.getDataCenterId());
+ }
}
- for (Cluster cluster : clusters) {
- if (cluster.getAllocationState() == AllocationState.Enabled) {
- List<HostVO> hosts = _hostDao.findByClusterId(cluster.getId());
+ return hostVO;
+ }
+
+ private HostVO getHost(Long zoneId, boolean computeClusterMustSupportResign) {
+ Preconditions.checkArgument(zoneId != null, "Zone ID cannot be null.");
- if (hosts != null) {
- for (HostVO host : hosts) {
- if (host.getResourceState() == ResourceState.Enabled) {
- return host;
- }
- }
+ List<HostVO> hosts = _hostDao.listByDataCenterIdAndHypervisorType(zoneId, HypervisorType.XenServer);
+
+ if (hosts == null) {
+ return null;
+ }
+
+ List<Long> clustersToSkip = new ArrayList<>();
+
+ Collections.shuffle(hosts, random);
+
+ for (HostVO host : hosts) {
+ if (computeClusterMustSupportResign) {
+ long clusterId = host.getClusterId();
+
+ if (clustersToSkip.contains(clusterId)) {
+ continue;
+ }
+
+ if (clusterDao.computeWhetherClusterSupportsResigning(clusterId)) {
+ return host;
+ }
+ else {
+ clustersToSkip.add(clusterId);
}
}
+ else {
+ return host;
+ }
}
- throw new CloudRuntimeException("Unable to locate an applicable cluster");
+ return null;
}
@Override
public void copyAsync(Map<VolumeInfo, DataStore> volumeMap, VirtualMachineTO vmTo, Host srcHost, Host destHost, AsyncCompletionCallback<CopyCommandResult> callback) {
CopyCommandResult result = new CopyCommandResult(null, null);
+
result.setResult("Unsupported operation requested for copying data.");
+
callback.complete(result);
}
+
+ private Map<String, String> getDetails(DataObject dataObj) {
+ if (dataObj instanceof VolumeInfo) {
+ return getVolumeDetails((VolumeInfo)dataObj);
+ }
+ else if (dataObj instanceof SnapshotInfo) {
+ return getSnapshotDetails((SnapshotInfo)dataObj);
+ }
+
+ throw new CloudRuntimeException("'dataObj' must be of type 'VolumeInfo' or 'SnapshotInfo'.");
+ }
+
+ private CopyCmdAnswer performResignature(DataObject dataObj, HostVO hostVO) {
+ return performResignature(dataObj, hostVO, false);
+ }
+
+ private CopyCmdAnswer performResignature(DataObject dataObj, HostVO hostVO, boolean keepGrantedAccess) {
+ long storagePoolId = dataObj.getDataStore().getId();
+ DataStore dataStore = dataStoreMgr.getDataStore(storagePoolId, DataStoreRole.Primary);
+
+ Map<String, String> details = getDetails(dataObj);
+
+ ResignatureCommand command = new ResignatureCommand(details);
+
+ ResignatureAnswer answer = null;
+
+ try {
+ _volumeService.grantAccess(dataObj, hostVO, dataStore);
+
+ answer = (ResignatureAnswer)_agentMgr.send(hostVO.getId(), command);
+ }
+ catch (AgentUnavailableException | OperationTimedoutException ex) {
+ keepGrantedAccess = false;
+
+ String msg = "Failed to revoke access to the volume from the hosts : ";
+
+ s_logger.warn(msg + ex.getMessage());
--- End diff --
Done
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by GabrielBrascher <gi...@git.apache.org>.
Github user GabrielBrascher commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r52108568
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -180,70 +208,119 @@ private Void handleCreateTemplateFromSnapshot(SnapshotInfo snapshotInfo, Templat
throw new CloudRuntimeException("This snapshot is not currently in a state where it can be used to create a template.");
}
- HostVO hostVO = getHost(snapshotInfo.getDataStore().getId());
- DataStore srcDataStore = snapshotInfo.getDataStore();
-
- String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
- int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
- CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), templateInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+ HostVO hostVO = getXenServerHost(snapshotInfo);
- String errMsg = null;
+ boolean usingBackendSnapshot = usingBackendSnapshotFor(snapshotInfo);
+ boolean computeClusterSupportsResign = computeClusterSupportsResign(hostVO.getClusterId());
- CopyCmdAnswer copyCmdAnswer = null;
+ if (usingBackendSnapshot && !computeClusterSupportsResign) {
+ throw new CloudRuntimeException("Unable to locate an applicable host with which to perform a resignature operation");
+ }
try {
- _volumeService.grantAccess(snapshotInfo, hostVO, srcDataStore);
+ if (usingBackendSnapshot) {
+ createVolumeFromSnapshot(hostVO, snapshotInfo, true);
+ }
- Map<String, String> srcDetails = getSnapshotDetails(_storagePoolDao.findById(srcDataStore.getId()), snapshotInfo);
+ DataStore srcDataStore = snapshotInfo.getDataStore();
- copyCommand.setOptions(srcDetails);
+ String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
+ int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
+ CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), templateInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+
+ String errMsg = null;
+
+ CopyCmdAnswer copyCmdAnswer = null;
- copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
- }
- catch (Exception ex) {
- throw new CloudRuntimeException(ex.getMessage());
- }
- finally {
try {
- _volumeService.revokeAccess(snapshotInfo, hostVO, srcDataStore);
+ // If we are using a back-end snapshot, then we should still have access to it from the hosts in the cluster that hostVO is in
+ // (because we passed in true as the third parameter to createVolumeFromSnapshot above).
+ if (usingBackendSnapshot == false) {
+ _volumeService.grantAccess(snapshotInfo, hostVO, srcDataStore);
+ }
+
+ Map<String, String> srcDetails = getSnapshotDetails(snapshotInfo);
+
+ copyCommand.setOptions(srcDetails);
+
+ copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
}
catch (Exception ex) {
- s_logger.debug(ex.getMessage(), ex);
+ throw new CloudRuntimeException(ex.getMessage());
}
-
- if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
- if (copyCmdAnswer != null && copyCmdAnswer.getDetails() != null && !copyCmdAnswer.getDetails().isEmpty()) {
- errMsg = copyCmdAnswer.getDetails();
+ finally {
+ try {
+ _volumeService.revokeAccess(snapshotInfo, hostVO, srcDataStore);
}
- else {
- errMsg = "Unable to perform host-side operation";
+ catch (Exception ex) {
+ s_logger.debug(ex.getMessage(), ex);
}
- }
- try {
- if (errMsg == null) {
- snapshotInfo.processEvent(Event.OperationSuccessed);
+ if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
+ if (copyCmdAnswer != null && copyCmdAnswer.getDetails() != null && !copyCmdAnswer.getDetails().isEmpty()) {
--- End diff --
@mike-tutkowski Could you please use StringUtils.isEmpty (https://commons.apache.org/proper/commons-lang/javadocs/api-2.6/org/apache/commons/lang/StringUtils.html#isEmpty(java.lang.String))?
This is a null safe test if the String **copyCmdAnswer.getDetails** is empty (you can use this method at lines 260, 363 and 403 -- in the **StorageSystemDataMotionStrategy** class).
Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by DaanHoogland <gi...@git.apache.org>.
Github user DaanHoogland commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r62880949
--- Diff: engine/storage/volume/src/org/apache/cloudstack/storage/volume/VolumeServiceImpl.java ---
@@ -134,6 +146,14 @@
EndPointSelector _epSelector;
@Inject
HostDao _hostDao;
+ @Inject
+ private PrimaryDataStoreDao _storagePoolDao;
+ @Inject
+ private HostDetailsDao _hostDetailsDao;
--- End diff --
not used
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by DaanHoogland <gi...@git.apache.org>.
Github user DaanHoogland commented on the pull request:
https://github.com/apache/cloudstack/pull/1403#issuecomment-218851513
@mike-tutkowski I don't mind ignoring most of the remaining comments I made except for
- the ones about returning null.
- the renamed exception, the old exception type seems more appropriate
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by DaanHoogland <gi...@git.apache.org>.
Github user DaanHoogland commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r63140242
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -361,59 +600,205 @@ private String getProperty(long snapshotId, String property) {
}
private Map<String, String> getVolumeDetails(VolumeInfo volumeInfo) {
- Map<String, String> sourceDetails = new HashMap<String, String>();
+ Map<String, String> volumeDetails = new HashMap<String, String>();
VolumeVO volumeVO = _volumeDao.findById(volumeInfo.getId());
long storagePoolId = volumeVO.getPoolId();
StoragePoolVO storagePoolVO = _storagePoolDao.findById(storagePoolId);
- sourceDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
- sourceDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
- sourceDetails.put(DiskTO.IQN, volumeVO.get_iScsiName());
+ volumeDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
+ volumeDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
+ volumeDetails.put(DiskTO.IQN, volumeVO.get_iScsiName());
ChapInfo chapInfo = _volumeService.getChapInfo(volumeInfo, volumeInfo.getDataStore());
if (chapInfo != null) {
- sourceDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, chapInfo.getInitiatorUsername());
- sourceDetails.put(DiskTO.CHAP_INITIATOR_SECRET, chapInfo.getInitiatorSecret());
- sourceDetails.put(DiskTO.CHAP_TARGET_USERNAME, chapInfo.getTargetUsername());
- sourceDetails.put(DiskTO.CHAP_TARGET_SECRET, chapInfo.getTargetSecret());
+ volumeDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, chapInfo.getInitiatorUsername());
+ volumeDetails.put(DiskTO.CHAP_INITIATOR_SECRET, chapInfo.getInitiatorSecret());
+ volumeDetails.put(DiskTO.CHAP_TARGET_USERNAME, chapInfo.getTargetUsername());
+ volumeDetails.put(DiskTO.CHAP_TARGET_SECRET, chapInfo.getTargetSecret());
}
- return sourceDetails;
+ return volumeDetails;
}
- public HostVO getHost(long dataStoreId) {
- StoragePoolVO storagePoolVO = _storagePoolDao.findById(dataStoreId);
+ private Map<String, String> getSnapshotDetails(SnapshotInfo snapshotInfo) {
+ Map<String, String> snapshotDetails = new HashMap<String, String>();
+
+ long storagePoolId = snapshotInfo.getDataStore().getId();
+ StoragePoolVO storagePoolVO = _storagePoolDao.findById(storagePoolId);
+
+ snapshotDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
+ snapshotDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
+
+ long snapshotId = snapshotInfo.getId();
+
+ snapshotDetails.put(DiskTO.IQN, getProperty(snapshotId, DiskTO.IQN));
- List<? extends Cluster> clusters = _mgr.searchForClusters(storagePoolVO.getDataCenterId(), new Long(0), Long.MAX_VALUE, HypervisorType.XenServer.toString());
+ snapshotDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, getProperty(snapshotId, DiskTO.CHAP_INITIATOR_USERNAME));
+ snapshotDetails.put(DiskTO.CHAP_INITIATOR_SECRET, getProperty(snapshotId, DiskTO.CHAP_INITIATOR_SECRET));
+ snapshotDetails.put(DiskTO.CHAP_TARGET_USERNAME, getProperty(snapshotId, DiskTO.CHAP_TARGET_USERNAME));
+ snapshotDetails.put(DiskTO.CHAP_TARGET_SECRET, getProperty(snapshotId, DiskTO.CHAP_TARGET_SECRET));
- if (clusters == null) {
- throw new CloudRuntimeException("Unable to locate an applicable cluster");
+ return snapshotDetails;
+ }
+
+ private HostVO getHost(SnapshotInfo snapshotInfo) {
+ HostVO hostVO = getHost(snapshotInfo.getDataCenterId(), true);
+
+ if (hostVO == null) {
+ hostVO = getHost(snapshotInfo.getDataCenterId(), false);
+
+ if (hostVO == null) {
+ throw new CloudRuntimeException("Unable to locate an applicable host in data center with ID = " + snapshotInfo.getDataCenterId());
+ }
}
- for (Cluster cluster : clusters) {
- if (cluster.getAllocationState() == AllocationState.Enabled) {
- List<HostVO> hosts = _hostDao.findByClusterId(cluster.getId());
+ return hostVO;
+ }
+
+ private HostVO getHost(Long zoneId, boolean computeClusterMustSupportResign) {
+ Preconditions.checkArgument(zoneId != null, "Zone ID cannot be null.");
- if (hosts != null) {
- for (HostVO host : hosts) {
- if (host.getResourceState() == ResourceState.Enabled) {
- return host;
- }
- }
+ List<HostVO> hosts = _hostDao.listByDataCenterIdAndHypervisorType(zoneId, HypervisorType.XenServer);
+
+ if (hosts == null) {
+ return null;
+ }
+
+ List<Long> clustersToSkip = new ArrayList<>();
+
+ Collections.shuffle(hosts, random);
+
+ for (HostVO host : hosts) {
+ if (computeClusterMustSupportResign) {
+ long clusterId = host.getClusterId();
+
+ if (clustersToSkip.contains(clusterId)) {
+ continue;
+ }
+
+ if (clusterDao.computeWhetherClusterSupportsResigning(clusterId)) {
+ return host;
+ }
+ else {
+ clustersToSkip.add(clusterId);
}
}
+ else {
+ return host;
+ }
}
- throw new CloudRuntimeException("Unable to locate an applicable cluster");
+ return null;
}
@Override
public void copyAsync(Map<VolumeInfo, DataStore> volumeMap, VirtualMachineTO vmTo, Host srcHost, Host destHost, AsyncCompletionCallback<CopyCommandResult> callback) {
CopyCommandResult result = new CopyCommandResult(null, null);
+
result.setResult("Unsupported operation requested for copying data.");
+
callback.complete(result);
}
+
+ private Map<String, String> getDetails(DataObject dataObj) {
+ if (dataObj instanceof VolumeInfo) {
+ return getVolumeDetails((VolumeInfo)dataObj);
+ }
+ else if (dataObj instanceof SnapshotInfo) {
+ return getSnapshotDetails((SnapshotInfo)dataObj);
+ }
+
+ throw new CloudRuntimeException("'dataObj' must be of type 'VolumeInfo' or 'SnapshotInfo'.");
+ }
+
+ private CopyCmdAnswer performResignature(DataObject dataObj, HostVO hostVO) {
+ return performResignature(dataObj, hostVO, false);
+ }
+
+ private CopyCmdAnswer performResignature(DataObject dataObj, HostVO hostVO, boolean keepGrantedAccess) {
+ long storagePoolId = dataObj.getDataStore().getId();
+ DataStore dataStore = dataStoreMgr.getDataStore(storagePoolId, DataStoreRole.Primary);
+
+ Map<String, String> details = getDetails(dataObj);
+
+ ResignatureCommand command = new ResignatureCommand(details);
+
+ ResignatureAnswer answer = null;
+
+ try {
+ _volumeService.grantAccess(dataObj, hostVO, dataStore);
+
+ answer = (ResignatureAnswer)_agentMgr.send(hostVO.getId(), command);
+ }
+ catch (AgentUnavailableException | OperationTimedoutException ex) {
+ keepGrantedAccess = false;
+
+ String msg = "Failed to revoke access to the volume from the hosts : ";
+
+ s_logger.warn(msg + ex.getMessage());
--- End diff --
@jburwell @mike-tutkowski I would like to see stacktraces at debug level and below but not at warning level. This is an operational level and should not contain such coding/debugging info.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r59074209
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -114,6 +142,17 @@ private boolean canHandle(DataStore dataStore) {
return true;
}
+ } else if (dataObject instanceof TemplateInfo) {
+ // If the storage system can clone volumes, we can cache templates on it.
+ String value = mapCapabilities.get(DataStoreCapabilities.CAN_CREATE_VOLUME_FROM_VOLUME.toString());
+ Boolean canCloneVolume = new Boolean(value);
--- End diff --
``new Boolean`` skips the constant pool -- placing unnecessary pressure on the garbage collector. Use ``Boolean.valueOf`` instead.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by rafaelweingartner <gi...@git.apache.org>.
Github user rafaelweingartner commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r59078361
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -172,78 +232,211 @@ private void validate(SnapshotInfo snapshotInfo) {
}
}
- private Void handleCreateTemplateFromSnapshot(SnapshotInfo snapshotInfo, TemplateInfo templateInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
+ private boolean usingBackendSnapshotFor(SnapshotInfo snapshotInfo) {
+ String property = getProperty(snapshotInfo.getId(), "takeSnapshot");
+
+ return Boolean.parseBoolean(property);
+ }
+
+ private void handleCreateTemplateFromSnapshot(SnapshotInfo snapshotInfo, TemplateInfo templateInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
try {
snapshotInfo.processEvent(Event.CopyingRequested);
}
catch (Exception ex) {
throw new CloudRuntimeException("This snapshot is not currently in a state where it can be used to create a template.");
}
- HostVO hostVO = getHost(snapshotInfo.getDataStore().getId());
- DataStore srcDataStore = snapshotInfo.getDataStore();
-
- String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
- int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
- CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), templateInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+ HostVO hostVO = getHost(snapshotInfo);
- String errMsg = null;
+ boolean usingBackendSnapshot = usingBackendSnapshotFor(snapshotInfo);
+ boolean computeClusterSupportsResign = computeClusterSupportsResign(hostVO.getClusterId());
- CopyCmdAnswer copyCmdAnswer = null;
+ if (usingBackendSnapshot && !computeClusterSupportsResign) {
+ throw new CloudRuntimeException("Unable to locate an applicable host with which to perform a resignature operation");
+ }
try {
- _volumeService.grantAccess(snapshotInfo, hostVO, srcDataStore);
+ if (usingBackendSnapshot) {
+ createVolumeFromSnapshot(hostVO, snapshotInfo, true);
+ }
- Map<String, String> srcDetails = getSnapshotDetails(_storagePoolDao.findById(srcDataStore.getId()), snapshotInfo);
+ DataStore srcDataStore = snapshotInfo.getDataStore();
- copyCommand.setOptions(srcDetails);
+ String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
+ int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
+ CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), templateInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+
+ String errMsg = null;
+
+ CopyCmdAnswer copyCmdAnswer = null;
- copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
- }
- catch (Exception ex) {
- throw new CloudRuntimeException(ex.getMessage());
- }
- finally {
try {
--- End diff --
I believe we can extract lines 273-288 to a method. Then, that block comment there can become a java doc. It would also enable the writing of a test case.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r59074444
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -132,31 +171,52 @@ public void copyAsync(DataObject srcData, DataObject destData, Host destHost, As
validate(snapshotInfo);
- boolean canHandleSrc = canHandle(srcData.getDataStore());
+ boolean canHandleSrc = canHandle(srcData);
if (canHandleSrc && destData instanceof TemplateInfo &&
(destData.getDataStore().getRole() == DataStoreRole.Image || destData.getDataStore().getRole() == DataStoreRole.ImageCache)) {
handleCreateTemplateFromSnapshot(snapshotInfo, (TemplateInfo)destData, callback);
+
return;
}
if (destData instanceof VolumeInfo) {
VolumeInfo volumeInfo = (VolumeInfo)destData;
- boolean canHandleDest = canHandle(destData.getDataStore());
+
+ boolean canHandleDest = canHandle(destData);
if (canHandleSrc && canHandleDest) {
- handleCreateVolumeFromSnapshotBothOnStorageSystem(snapshotInfo, volumeInfo, callback);
- return;
+ if (snapshotInfo.getDataStore().getId() == volumeInfo.getDataStore().getId()) {
+ handleCreateVolumeFromSnapshotBothOnStorageSystem(snapshotInfo, volumeInfo, callback);
+ return;
+ }
+ else {
+ throw new UnsupportedOperationException("This operation is not supported (DataStoreCapabilities.STORAGE_SYSTEM_SNAPSHOT " +
+ "not supported by source or destination storage plug-in).");
+ }
}
+
if (canHandleSrc) {
throw new UnsupportedOperationException("This operation is not supported (DataStoreCapabilities.STORAGE_SYSTEM_SNAPSHOT " +
"not supported by destination storage plug-in).");
}
+
if (canHandleDest) {
throw new UnsupportedOperationException("This operation is not supported (DataStoreCapabilities.STORAGE_SYSTEM_SNAPSHOT " +
"not supported by source storage plug-in).");
}
}
+ } else if (srcData instanceof TemplateInfo && destData instanceof VolumeInfo) {
+ boolean canHandleSrc = canHandle(srcData);
+
+ if (!canHandleSrc) {
+ throw new UnsupportedOperationException("This operation is not supported (DataStoreCapabilities.STORAGE_CAN_CREATE_VOLUME_FROM_VOLUME " +
--- End diff --
To aid debugging, please include the name of the destination storage plugin.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r59078926
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -172,78 +232,211 @@ private void validate(SnapshotInfo snapshotInfo) {
}
}
- private Void handleCreateTemplateFromSnapshot(SnapshotInfo snapshotInfo, TemplateInfo templateInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
+ private boolean usingBackendSnapshotFor(SnapshotInfo snapshotInfo) {
+ String property = getProperty(snapshotInfo.getId(), "takeSnapshot");
+
+ return Boolean.parseBoolean(property);
+ }
+
+ private void handleCreateTemplateFromSnapshot(SnapshotInfo snapshotInfo, TemplateInfo templateInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
try {
snapshotInfo.processEvent(Event.CopyingRequested);
}
catch (Exception ex) {
throw new CloudRuntimeException("This snapshot is not currently in a state where it can be used to create a template.");
}
- HostVO hostVO = getHost(snapshotInfo.getDataStore().getId());
- DataStore srcDataStore = snapshotInfo.getDataStore();
-
- String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
- int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
- CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), templateInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+ HostVO hostVO = getHost(snapshotInfo);
- String errMsg = null;
+ boolean usingBackendSnapshot = usingBackendSnapshotFor(snapshotInfo);
+ boolean computeClusterSupportsResign = computeClusterSupportsResign(hostVO.getClusterId());
- CopyCmdAnswer copyCmdAnswer = null;
+ if (usingBackendSnapshot && !computeClusterSupportsResign) {
+ throw new CloudRuntimeException("Unable to locate an applicable host with which to perform a resignature operation");
+ }
try {
- _volumeService.grantAccess(snapshotInfo, hostVO, srcDataStore);
+ if (usingBackendSnapshot) {
+ createVolumeFromSnapshot(hostVO, snapshotInfo, true);
+ }
- Map<String, String> srcDetails = getSnapshotDetails(_storagePoolDao.findById(srcDataStore.getId()), snapshotInfo);
+ DataStore srcDataStore = snapshotInfo.getDataStore();
- copyCommand.setOptions(srcDetails);
+ String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
+ int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
+ CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), templateInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+
+ String errMsg = null;
+
+ CopyCmdAnswer copyCmdAnswer = null;
- copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
- }
- catch (Exception ex) {
- throw new CloudRuntimeException(ex.getMessage());
- }
- finally {
try {
- _volumeService.revokeAccess(snapshotInfo, hostVO, srcDataStore);
+ // If we are using a back-end snapshot, then we should still have access to it from the hosts in the cluster that hostVO is in
+ // (because we passed in true as the third parameter to createVolumeFromSnapshot above).
+ if (usingBackendSnapshot == false) {
+ _volumeService.grantAccess(snapshotInfo, hostVO, srcDataStore);
+ }
+
+ Map<String, String> srcDetails = getSnapshotDetails(snapshotInfo);
+
+ copyCommand.setOptions(srcDetails);
+
+ copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
}
catch (Exception ex) {
- s_logger.debug(ex.getMessage(), ex);
+ throw new CloudRuntimeException(ex.getMessage());
}
+ finally {
+ try {
+ _volumeService.revokeAccess(snapshotInfo, hostVO, srcDataStore);
+ }
+ catch (Exception ex) {
+ s_logger.debug(ex.getMessage(), ex);
+ }
- if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
- if (copyCmdAnswer != null && copyCmdAnswer.getDetails() != null && !copyCmdAnswer.getDetails().isEmpty()) {
- errMsg = copyCmdAnswer.getDetails();
+ if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
+ if (copyCmdAnswer != null && !StringUtils.isEmpty(copyCmdAnswer.getDetails())) {
+ errMsg = copyCmdAnswer.getDetails();
+ }
+ else {
+ errMsg = "Unable to perform host-side operation";
+ }
}
- else {
- errMsg = "Unable to perform host-side operation";
+
+ try {
+ if (errMsg == null) {
+ snapshotInfo.processEvent(Event.OperationSuccessed);
+ }
+ else {
+ snapshotInfo.processEvent(Event.OperationFailed);
+ }
+ }
+ catch (Exception ex) {
+ s_logger.debug(ex.getMessage(), ex);
}
}
- try {
- if (errMsg == null) {
- snapshotInfo.processEvent(Event.OperationSuccessed);
+ CopyCommandResult result = new CopyCommandResult(null, copyCmdAnswer);
+
+ result.setResult(errMsg);
+
+ callback.complete(result);
+ }
+ finally {
+ if (usingBackendSnapshot) {
+ deleteVolumeFromSnapshot(snapshotInfo);
+ }
+ }
+ }
+
+ /**
+ * Clones a template present on the storage to a new volume and resignatures it.
+ *
+ * @param srcData source template
+ * @param destData destination ROOT volume
+ * @param callback for async
+ */
+ private void handleCreateVolumeFromTemplateBothOnStorageSystem(DataObject srcData, DataObject destData, AsyncCompletionCallback<CopyCommandResult> callback) {
+ CopyCmdAnswer copyCmdAnswer = null;
+ String errMsg = null;
+
+ TemplateInfo templateInfo = (TemplateInfo)srcData;
+ VolumeInfo volumeInfo = (VolumeInfo)destData;
+
+ HostVO hostVO = getHost(volumeInfo.getDataCenterId(), true);
+
+ if (hostVO == null) {
+ throw new CloudRuntimeException("Unable to locate a host capable of resigning in the zone with the following ID: " + volumeInfo.getDataCenterId());
+ }
+
+ boolean computeClusterSupportsResign = computeClusterSupportsResign(hostVO.getClusterId());
+
+ if (!computeClusterSupportsResign) {
+ throw new CloudRuntimeException("Unable to locate an applicable host with which to perform a resignature operation");
+ }
+
+ try {
+ VolumeDetailVO volumeDetail = new VolumeDetailVO(volumeInfo.getId(),
+ "cloneOfTemplate",
+ String.valueOf(templateInfo.getId()),
+ false);
+
+ volumeDetail = _volumeDetailsDao.persist(volumeDetail);
+
+ AsyncCallFuture<VolumeApiResult> future = _volumeService.createVolumeAsync(volumeInfo, volumeInfo.getDataStore());
+ VolumeApiResult result = future.get();
+
+ if (volumeDetail != null) {
+ _volumeDetailsDao.remove(volumeDetail.getId());
+ }
+
+ if (result.isFailed()) {
+ s_logger.debug("Failed to create a volume: " + result.getResult());
+
+ throw new CloudRuntimeException(result.getResult());
+ }
+
+ volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
+
+ volumeInfo.processEvent(Event.MigrationRequested);
+
+ volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
+
+ copyCmdAnswer = performResignature(volumeInfo, hostVO);
+
+ if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
+ if (copyCmdAnswer != null && !StringUtils.isEmpty(copyCmdAnswer.getDetails())) {
+ throw new CloudRuntimeException(copyCmdAnswer.getDetails());
}
else {
- snapshotInfo.processEvent(Event.OperationFailed);
+ throw new CloudRuntimeException("Unable to perform host-side operation");
}
}
- catch (Exception ex) {
- s_logger.debug(ex.getMessage(), ex);
+ } catch (Exception ex) {
+ try {
+ volumeInfo.getDataStore().getDriver().deleteAsync(volumeInfo.getDataStore(), volumeInfo, null);
+ } catch (Exception e) {
+ s_logger.debug("Exception while deleting volume on the backend " + e.getMessage());
}
+
+ throw new CloudRuntimeException("Create volume from template failed " + ex.getMessage());
--- End diff --
Please add the template ID to the error message to assist operational debugging.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by GabrielBrascher <gi...@git.apache.org>.
Github user GabrielBrascher commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r52108928
--- Diff: plugins/hypervisors/xenserver/src/com/cloud/hypervisor/xenserver/resource/CitrixResourceBase.java ---
@@ -168,7 +168,9 @@
public abstract class CitrixResourceBase implements ServerResource, HypervisorResource, VirtualRouterDeployer {
public enum SRType {
- EXT, FILE, ISCSI, ISO, LVM, LVMOHBA, LVMOISCSI, NFS;
+ // RELVMOISCSI = used for resigning metadata (like SR UUID and VDI UUID when a
+ // particular storage manager is installed on a XenServer host (for back-end snapshots to work))
+ EXT, FILE, ISCSI, ISO, LVM, LVMOHBA, LVMOISCSI, RELVMOISCSI, NFS;
--- End diff --
@mike-tutkowski Sorry if I am being too repetitive.
As an idea, this comments might serve well as a Javadoc block documenting the enum class. If you are friendly with this idea, the same could be done with the **org.apache.cloudstack.engine.subsystem.api.storage.DataStoreCapabilities** enum class.
Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by mike-tutkowski <gi...@git.apache.org>.
Github user mike-tutkowski commented on the pull request:
https://github.com/apache/cloudstack/pull/1403#issuecomment-207535756
Here are my updated Marvin tests for this feature:
[TestSnapshots.py.txt](https://github.com/apache/cloudstack/files/210618/TestSnapshots.py.txt)
[TestVMSnapshots.py.txt](https://github.com/apache/cloudstack/files/210617/TestVMSnapshots.py.txt)
[TestVolumes.py.txt](https://github.com/apache/cloudstack/files/210619/TestVolumes.py.txt)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r59078225
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -255,99 +448,153 @@ private Void handleCreateVolumeFromSnapshotBothOnStorageSystem(SnapshotInfo snap
VolumeApiResult result = future.get();
+ if (volumeDetail != null) {
+ _volumeDetailsDao.remove(volumeDetail.getId());
+ }
+
if (result.isFailed()) {
s_logger.debug("Failed to create a volume: " + result.getResult());
throw new CloudRuntimeException(result.getResult());
}
- }
- catch (Exception ex) {
- throw new CloudRuntimeException(ex.getMessage());
- }
- volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
+ volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
- volumeInfo.processEvent(Event.MigrationRequested);
+ volumeInfo.processEvent(Event.MigrationRequested);
- volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
+ volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
- HostVO hostVO = getHost(snapshotInfo.getDataStore().getId());
+ if (useCloning) {
+ copyCmdAnswer = performResignature(volumeInfo, hostVO);
+ }
+ else {
+ // asking for a XenServer host here so we don't always prefer to use XenServer hosts that support resigning
+ // even when we don't need those hosts to do this kind of copy work
+ hostVO = getHost(snapshotInfo.getDataCenterId(), false);
- String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
- int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
- CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), volumeInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+ copyCmdAnswer = performCopyOfVdi(volumeInfo, snapshotInfo, hostVO);
+ }
- CopyCmdAnswer copyCmdAnswer = null;
+ if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
+ if (copyCmdAnswer != null && !StringUtils.isEmpty(copyCmdAnswer.getDetails())) {
+ errMsg = copyCmdAnswer.getDetails();
+ }
+ else {
+ errMsg = "Unable to perform host-side operation";
+ }
+ }
+ }
+ catch (Exception ex) {
+ errMsg = ex.getMessage() != null ? ex.getMessage() : "Copy operation failed";
+ }
- try {
- _volumeService.grantAccess(snapshotInfo, hostVO, snapshotInfo.getDataStore());
- _volumeService.grantAccess(volumeInfo, hostVO, volumeInfo.getDataStore());
+ CopyCommandResult result = new CopyCommandResult(null, copyCmdAnswer);
- Map<String, String> srcDetails = getSnapshotDetails(_storagePoolDao.findById(snapshotInfo.getDataStore().getId()), snapshotInfo);
+ result.setResult(errMsg);
- copyCommand.setOptions(srcDetails);
+ callback.complete(result);
+ }
- Map<String, String> destDetails = getVolumeDetails(volumeInfo);
+ /**
+ * If the underlying storage system is making use of read-only snapshots, this gives the storage system the opportunity to
+ * create a volume from the snapshot so that we can copy the VHD file that should be inside of the snapshot to secondary storage.
+ *
+ * The resultant volume must be writable because we need to resign the SR and the VDI that should be inside of it before we copy
+ * the VHD file to secondary storage.
+ *
+ * If the storage system is using writable snapshots, then nothing need be done by that storage system here because we can just
+ * resign the SR and the VDI that should be inside of the snapshot before copying the VHD file to secondary storage.
+ */
+ private void createVolumeFromSnapshot(HostVO hostVO, SnapshotInfo snapshotInfo, boolean keepGrantedAccess) {
+ SnapshotDetailsVO snapshotDetails = handleSnapshotDetails(snapshotInfo.getId(), "tempVolume", "create");
- copyCommand.setOptions2(destDetails);
-
- copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
- }
- catch (Exception ex) {
- throw new CloudRuntimeException(ex.getMessage());
+ try {
+ snapshotInfo.getDataStore().getDriver().createAsync(snapshotInfo.getDataStore(), snapshotInfo, null);
}
finally {
- try {
- _volumeService.revokeAccess(snapshotInfo, hostVO, snapshotInfo.getDataStore());
- }
- catch (Exception ex) {
- s_logger.debug(ex.getMessage(), ex);
- }
-
- try {
- _volumeService.revokeAccess(volumeInfo, hostVO, volumeInfo.getDataStore());
- }
- catch (Exception ex) {
- s_logger.debug(ex.getMessage(), ex);
- }
+ _snapshotDetailsDao.remove(snapshotDetails.getId());
}
- String errMsg = null;
+ CopyCmdAnswer copyCmdAnswer = performResignature(snapshotInfo, hostVO, keepGrantedAccess);
if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
- if (copyCmdAnswer != null && copyCmdAnswer.getDetails() != null && !copyCmdAnswer.getDetails().isEmpty()) {
- errMsg = copyCmdAnswer.getDetails();
+ if (copyCmdAnswer != null && !StringUtils.isEmpty(copyCmdAnswer.getDetails())) {
+ throw new CloudRuntimeException(copyCmdAnswer.getDetails());
}
else {
- errMsg = "Unable to perform host-side operation";
+ throw new CloudRuntimeException("Unable to perform host-side operation");
}
}
+ }
- CopyCommandResult result = new CopyCommandResult(null, copyCmdAnswer);
+ /**
+ * If the underlying storage system needed to create a volume from a snapshot for createVolumeFromSnapshot(HostVO, SnapshotInfo), then
+ * this is its opportunity to delete that temporary volume and restore properties in snapshot_details to the way they were before the
+ * invocation of createVolumeFromSnapshot(HostVO, SnapshotInfo).
+ */
+ private void deleteVolumeFromSnapshot(SnapshotInfo snapshotInfo) {
+ SnapshotDetailsVO snapshotDetails = handleSnapshotDetails(snapshotInfo.getId(), "tempVolume", "delete");
- result.setResult(errMsg);
+ try {
+ snapshotInfo.getDataStore().getDriver().createAsync(snapshotInfo.getDataStore(), snapshotInfo, null);
+ }
+ finally {
+ _snapshotDetailsDao.remove(snapshotDetails.getId());
+ }
+ }
- callback.complete(result);
+ private SnapshotDetailsVO handleSnapshotDetails(long csSnapshotId, String name, String value) {
+ _snapshotDetailsDao.removeDetail(csSnapshotId, name);
- return null;
+ SnapshotDetailsVO snapshotDetails = new SnapshotDetailsVO(csSnapshotId, name, value, false);
+
+ return _snapshotDetailsDao.persist(snapshotDetails);
}
- private Map<String, String> getSnapshotDetails(StoragePoolVO storagePoolVO, SnapshotInfo snapshotInfo) {
- Map<String, String> details = new HashMap<String, String>();
+ private boolean canStorageSystemCreateVolumeFromVolume(SnapshotInfo snapshotInfo) {
+ boolean supportsCloningVolumeFromVolume = false;
- details.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
- details.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
+ DataStore dataStore = _dataStoreMgr.getDataStore(snapshotInfo.getDataStore().getId(), DataStoreRole.Primary);
- long snapshotId = snapshotInfo.getId();
+ Map<String, String> mapCapabilities = dataStore.getDriver().getCapabilities();
- details.put(DiskTO.IQN, getProperty(snapshotId, DiskTO.IQN));
+ if (mapCapabilities != null) {
+ String value = mapCapabilities.get(DataStoreCapabilities.CAN_CREATE_VOLUME_FROM_VOLUME.toString());
- details.put(DiskTO.CHAP_INITIATOR_USERNAME, getProperty(snapshotId, DiskTO.CHAP_INITIATOR_USERNAME));
- details.put(DiskTO.CHAP_INITIATOR_SECRET, getProperty(snapshotId, DiskTO.CHAP_INITIATOR_SECRET));
- details.put(DiskTO.CHAP_TARGET_USERNAME, getProperty(snapshotId, DiskTO.CHAP_TARGET_USERNAME));
- details.put(DiskTO.CHAP_TARGET_SECRET, getProperty(snapshotId, DiskTO.CHAP_TARGET_SECRET));
+ supportsCloningVolumeFromVolume = new Boolean(value);
+ }
- return details;
+ return supportsCloningVolumeFromVolume;
+ }
+
+ private boolean computeClusterSupportsResign(long clusterId) {
+ List<HostVO> hosts = _hostDao.findByClusterId(clusterId);
+
+ if (hosts == null) {
+ return false;
+ }
+
+ for (HostVO host : hosts) {
+ if (host == null) {
+ return false;
+ }
+
+ DetailVO hostDetail = _hostDetailsDao.findDetail(host.getId(), "supportsResign");
+
+ if (hostDetail == null) {
+ return false;
+ }
+
+ String value = hostDetail.getValue();
+
+ Boolean booleanValue = Boolean.valueOf(value);
+
+ if (booleanValue == false) {
+ return false;
+ }
+ }
--- End diff --
Would it be possible to put the logic of this entire method into a single query and expose it as a method on ``ClusterDAO`` or ``HostDetailsDAO``? As this method is written, the performance will vary greatly based on the cluster and whether or not there is a host that supports resign. A single join would make the performance of this essentially the same regardless of the cluster size or the presence of a host that supports resign.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r59084839
--- Diff: engine/storage/snapshot/src/org/apache/cloudstack/storage/snapshot/StorageSystemSnapshotStrategy.java ---
@@ -387,38 +424,87 @@ private String getProperty(long snapshotId, String property) {
return null;
}
- private HostVO getHost(Long hostId, VolumeVO volumeVO) {
- HostVO hostVO = _hostDao.findById(hostId);
+ private HostVO getHost(long volumeId) {
+ VolumeVO volumeVO = _volumeDao.findById(volumeId);
+
+ Long vmInstanceId = volumeVO.getInstanceId();
+ VMInstanceVO vmInstanceVO = _vmInstanceDao.findById(vmInstanceId);
+
+ Long hostId = null;
+
+ // if the volume to snapshot is associated with a VM
+ if (vmInstanceVO != null) {
+ hostId = vmInstanceVO.getHostId();
+
+ // if the VM is not associated with a host
+ if (hostId == null) {
+ hostId = vmInstanceVO.getLastHostId();
+ }
+ }
+
+ return getHost(volumeVO.getDataCenterId(), hostId);
+ }
+
+ private HostVO getHost(long zoneId, Long hostId) {
+ HostVO hostVO = getHost(zoneId, true);
if (hostVO != null) {
return hostVO;
}
- // pick a host in any XenServer cluster that's in the applicable zone
+ hostVO = _hostDao.findById(hostId);
- long zoneId = volumeVO.getDataCenterId();
+ if (hostVO != null) {
+ return hostVO;
+ }
+
+ hostVO = getHost(zoneId, false);
+
+ if (hostVO != null) {
+ return hostVO;
+ }
- List<? extends Cluster> clusters = _mgr.searchForClusters(zoneId, new Long(0), Long.MAX_VALUE, HypervisorType.XenServer.toString());
+ throw new CloudRuntimeException("Unable to locate an applicable host");
+ }
+
+ private HostVO getHost(long zoneId, boolean computeClusterMustSupportResign) {
+ List<? extends Cluster> clusters = _mgr.searchForClusters(zoneId, 0L, Long.MAX_VALUE, HypervisorType.XenServer.toString());
if (clusters == null) {
- throw new CloudRuntimeException("Unable to locate an applicable cluster");
+ clusters = new ArrayList<>();
}
+ Collections.shuffle(clusters, new Random(System.nanoTime()));
+
+ clusters:
for (Cluster cluster : clusters) {
if (cluster.getAllocationState() == AllocationState.Enabled) {
List<HostVO> hosts = _hostDao.findByClusterId(cluster.getId());
if (hosts != null) {
+ Collections.shuffle(hosts, new Random(System.nanoTime()));
+
for (HostVO host : hosts) {
if (host.getResourceState() == ResourceState.Enabled) {
- return host;
+ if (computeClusterMustSupportResign) {
+ if (computeClusterSupportsResign(cluster.getId())) {
+ return host;
+ }
+ else {
+ // no other host in the cluster in question should be able to satisfy our requirements here, so move on to the next cluster
+ continue clusters;
+ }
+ }
+ else {
+ return host;
+ }
}
}
}
}
}
--- End diff --
**I realize this issue predates this patch. Therefore, I understand if it is not possible to address it now.**
This method appears to be a duplicate of another implementation. Would be possible to consolidate them for consistency and maintainability?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by mike-tutkowski <gi...@git.apache.org>.
Github user mike-tutkowski commented on the pull request:
https://github.com/apache/cloudstack/pull/1403#issuecomment-207555843
The CloudStack community can definitely have those tests, if it wants them.
Right now, however, we don't have any way to enable other people (besides those who own SolidFire equipment) to successfully run those tests. Perhaps SolidFire will have a virtual node at some point that can be handed over to Apache Infra for this purpose.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r59075182
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -172,78 +232,211 @@ private void validate(SnapshotInfo snapshotInfo) {
}
}
- private Void handleCreateTemplateFromSnapshot(SnapshotInfo snapshotInfo, TemplateInfo templateInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
+ private boolean usingBackendSnapshotFor(SnapshotInfo snapshotInfo) {
+ String property = getProperty(snapshotInfo.getId(), "takeSnapshot");
+
+ return Boolean.parseBoolean(property);
+ }
+
+ private void handleCreateTemplateFromSnapshot(SnapshotInfo snapshotInfo, TemplateInfo templateInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
try {
snapshotInfo.processEvent(Event.CopyingRequested);
}
catch (Exception ex) {
throw new CloudRuntimeException("This snapshot is not currently in a state where it can be used to create a template.");
}
- HostVO hostVO = getHost(snapshotInfo.getDataStore().getId());
- DataStore srcDataStore = snapshotInfo.getDataStore();
-
- String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
- int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
- CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), templateInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+ HostVO hostVO = getHost(snapshotInfo);
- String errMsg = null;
+ boolean usingBackendSnapshot = usingBackendSnapshotFor(snapshotInfo);
+ boolean computeClusterSupportsResign = computeClusterSupportsResign(hostVO.getClusterId());
- CopyCmdAnswer copyCmdAnswer = null;
+ if (usingBackendSnapshot && !computeClusterSupportsResign) {
+ throw new CloudRuntimeException("Unable to locate an applicable host with which to perform a resignature operation");
+ }
try {
- _volumeService.grantAccess(snapshotInfo, hostVO, srcDataStore);
+ if (usingBackendSnapshot) {
+ createVolumeFromSnapshot(hostVO, snapshotInfo, true);
+ }
- Map<String, String> srcDetails = getSnapshotDetails(_storagePoolDao.findById(srcDataStore.getId()), snapshotInfo);
+ DataStore srcDataStore = snapshotInfo.getDataStore();
- copyCommand.setOptions(srcDetails);
+ String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
+ int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
+ CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), templateInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+
+ String errMsg = null;
+
+ CopyCmdAnswer copyCmdAnswer = null;
- copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
- }
- catch (Exception ex) {
- throw new CloudRuntimeException(ex.getMessage());
- }
- finally {
try {
- _volumeService.revokeAccess(snapshotInfo, hostVO, srcDataStore);
+ // If we are using a back-end snapshot, then we should still have access to it from the hosts in the cluster that hostVO is in
+ // (because we passed in true as the third parameter to createVolumeFromSnapshot above).
+ if (usingBackendSnapshot == false) {
+ _volumeService.grantAccess(snapshotInfo, hostVO, srcDataStore);
+ }
+
+ Map<String, String> srcDetails = getSnapshotDetails(snapshotInfo);
+
+ copyCommand.setOptions(srcDetails);
+
+ copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
}
catch (Exception ex) {
- s_logger.debug(ex.getMessage(), ex);
+ throw new CloudRuntimeException(ex.getMessage());
}
+ finally {
+ try {
+ _volumeService.revokeAccess(snapshotInfo, hostVO, srcDataStore);
+ }
+ catch (Exception ex) {
+ s_logger.debug(ex.getMessage(), ex);
+ }
- if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
- if (copyCmdAnswer != null && copyCmdAnswer.getDetails() != null && !copyCmdAnswer.getDetails().isEmpty()) {
- errMsg = copyCmdAnswer.getDetails();
+ if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
+ if (copyCmdAnswer != null && !StringUtils.isEmpty(copyCmdAnswer.getDetails())) {
+ errMsg = copyCmdAnswer.getDetails();
+ }
+ else {
+ errMsg = "Unable to perform host-side operation";
+ }
}
- else {
- errMsg = "Unable to perform host-side operation";
+
+ try {
+ if (errMsg == null) {
--- End diff --
Consider using ``Strings.isNullOrEmpty`` to check the value ``errMsg`` to cover both the ``null`` and blank cases.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r59079506
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -361,59 +608,225 @@ private String getProperty(long snapshotId, String property) {
}
private Map<String, String> getVolumeDetails(VolumeInfo volumeInfo) {
- Map<String, String> sourceDetails = new HashMap<String, String>();
+ Map<String, String> volumeDetails = new HashMap<String, String>();
VolumeVO volumeVO = _volumeDao.findById(volumeInfo.getId());
long storagePoolId = volumeVO.getPoolId();
StoragePoolVO storagePoolVO = _storagePoolDao.findById(storagePoolId);
- sourceDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
- sourceDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
- sourceDetails.put(DiskTO.IQN, volumeVO.get_iScsiName());
+ volumeDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
+ volumeDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
+ volumeDetails.put(DiskTO.IQN, volumeVO.get_iScsiName());
ChapInfo chapInfo = _volumeService.getChapInfo(volumeInfo, volumeInfo.getDataStore());
if (chapInfo != null) {
- sourceDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, chapInfo.getInitiatorUsername());
- sourceDetails.put(DiskTO.CHAP_INITIATOR_SECRET, chapInfo.getInitiatorSecret());
- sourceDetails.put(DiskTO.CHAP_TARGET_USERNAME, chapInfo.getTargetUsername());
- sourceDetails.put(DiskTO.CHAP_TARGET_SECRET, chapInfo.getTargetSecret());
+ volumeDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, chapInfo.getInitiatorUsername());
+ volumeDetails.put(DiskTO.CHAP_INITIATOR_SECRET, chapInfo.getInitiatorSecret());
+ volumeDetails.put(DiskTO.CHAP_TARGET_USERNAME, chapInfo.getTargetUsername());
+ volumeDetails.put(DiskTO.CHAP_TARGET_SECRET, chapInfo.getTargetSecret());
+ }
+
+ return volumeDetails;
+ }
+
+ private Map<String, String> getSnapshotDetails(SnapshotInfo snapshotInfo) {
+ Map<String, String> snapshotDetails = new HashMap<String, String>();
+
+ long storagePoolId = snapshotInfo.getDataStore().getId();
+ StoragePoolVO storagePoolVO = _storagePoolDao.findById(storagePoolId);
+
+ snapshotDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
+ snapshotDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
+
+ long snapshotId = snapshotInfo.getId();
+
+ snapshotDetails.put(DiskTO.IQN, getProperty(snapshotId, DiskTO.IQN));
+
+ snapshotDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, getProperty(snapshotId, DiskTO.CHAP_INITIATOR_USERNAME));
+ snapshotDetails.put(DiskTO.CHAP_INITIATOR_SECRET, getProperty(snapshotId, DiskTO.CHAP_INITIATOR_SECRET));
+ snapshotDetails.put(DiskTO.CHAP_TARGET_USERNAME, getProperty(snapshotId, DiskTO.CHAP_TARGET_USERNAME));
+ snapshotDetails.put(DiskTO.CHAP_TARGET_SECRET, getProperty(snapshotId, DiskTO.CHAP_TARGET_SECRET));
+
+ return snapshotDetails;
+ }
+
+ private HostVO getHost(SnapshotInfo snapshotInfo) {
+ HostVO hostVO = getHost(snapshotInfo.getDataCenterId(), true);
+
+ if (hostVO == null) {
+ hostVO = getHost(snapshotInfo.getDataCenterId(), false);
+
+ if (hostVO == null) {
+ throw new CloudRuntimeException("Unable to locate an applicable host");
+ }
}
- return sourceDetails;
+ return hostVO;
}
- public HostVO getHost(long dataStoreId) {
- StoragePoolVO storagePoolVO = _storagePoolDao.findById(dataStoreId);
+ private HostVO getHost(Long zoneId, boolean computeClusterMustSupportResign) {
+ if (zoneId == null) {
+ throw new CloudRuntimeException("Zone ID cannot be null.");
+ }
- List<? extends Cluster> clusters = _mgr.searchForClusters(storagePoolVO.getDataCenterId(), new Long(0), Long.MAX_VALUE, HypervisorType.XenServer.toString());
+ List<? extends Cluster> clusters = _mgr.searchForClusters(zoneId, new Long(0), Long.MAX_VALUE, HypervisorType.XenServer.toString());
if (clusters == null) {
- throw new CloudRuntimeException("Unable to locate an applicable cluster");
+ clusters = new ArrayList<>();
}
+ Collections.shuffle(clusters, new Random(System.nanoTime()));
--- End diff --
``new Random`` is expensive. Did you consider initializing a ``Random`` instance as a static member passing the same instance into ``shuffle``? Since ``Random`` is [thread-safe](http://bugs.java.com/bugdatabase/view_bug.do?bug_id=6362070), the same instance could be safely used. This would avoid the cost of seeding a RNG on every method call.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by DaanHoogland <gi...@git.apache.org>.
Github user DaanHoogland commented on the pull request:
https://github.com/apache/cloudstack/pull/1403#issuecomment-219161020
Then log the warning with just the message and add a debug statement with the stacktrace. logging stacktraces at a level even more strict then INFO does not compute.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r59076683
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -172,78 +232,211 @@ private void validate(SnapshotInfo snapshotInfo) {
}
}
- private Void handleCreateTemplateFromSnapshot(SnapshotInfo snapshotInfo, TemplateInfo templateInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
+ private boolean usingBackendSnapshotFor(SnapshotInfo snapshotInfo) {
+ String property = getProperty(snapshotInfo.getId(), "takeSnapshot");
+
+ return Boolean.parseBoolean(property);
+ }
+
+ private void handleCreateTemplateFromSnapshot(SnapshotInfo snapshotInfo, TemplateInfo templateInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
try {
snapshotInfo.processEvent(Event.CopyingRequested);
}
catch (Exception ex) {
throw new CloudRuntimeException("This snapshot is not currently in a state where it can be used to create a template.");
}
- HostVO hostVO = getHost(snapshotInfo.getDataStore().getId());
- DataStore srcDataStore = snapshotInfo.getDataStore();
-
- String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
- int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
- CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), templateInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+ HostVO hostVO = getHost(snapshotInfo);
- String errMsg = null;
+ boolean usingBackendSnapshot = usingBackendSnapshotFor(snapshotInfo);
+ boolean computeClusterSupportsResign = computeClusterSupportsResign(hostVO.getClusterId());
- CopyCmdAnswer copyCmdAnswer = null;
+ if (usingBackendSnapshot && !computeClusterSupportsResign) {
+ throw new CloudRuntimeException("Unable to locate an applicable host with which to perform a resignature operation");
+ }
try {
- _volumeService.grantAccess(snapshotInfo, hostVO, srcDataStore);
+ if (usingBackendSnapshot) {
+ createVolumeFromSnapshot(hostVO, snapshotInfo, true);
+ }
- Map<String, String> srcDetails = getSnapshotDetails(_storagePoolDao.findById(srcDataStore.getId()), snapshotInfo);
+ DataStore srcDataStore = snapshotInfo.getDataStore();
- copyCommand.setOptions(srcDetails);
+ String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
+ int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
+ CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), templateInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+
+ String errMsg = null;
+
+ CopyCmdAnswer copyCmdAnswer = null;
- copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
- }
- catch (Exception ex) {
- throw new CloudRuntimeException(ex.getMessage());
- }
- finally {
try {
- _volumeService.revokeAccess(snapshotInfo, hostVO, srcDataStore);
+ // If we are using a back-end snapshot, then we should still have access to it from the hosts in the cluster that hostVO is in
+ // (because we passed in true as the third parameter to createVolumeFromSnapshot above).
+ if (usingBackendSnapshot == false) {
+ _volumeService.grantAccess(snapshotInfo, hostVO, srcDataStore);
+ }
+
+ Map<String, String> srcDetails = getSnapshotDetails(snapshotInfo);
+
+ copyCommand.setOptions(srcDetails);
+
+ copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
}
catch (Exception ex) {
- s_logger.debug(ex.getMessage(), ex);
+ throw new CloudRuntimeException(ex.getMessage());
}
+ finally {
+ try {
+ _volumeService.revokeAccess(snapshotInfo, hostVO, srcDataStore);
+ }
+ catch (Exception ex) {
+ s_logger.debug(ex.getMessage(), ex);
+ }
- if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
- if (copyCmdAnswer != null && copyCmdAnswer.getDetails() != null && !copyCmdAnswer.getDetails().isEmpty()) {
- errMsg = copyCmdAnswer.getDetails();
+ if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
+ if (copyCmdAnswer != null && !StringUtils.isEmpty(copyCmdAnswer.getDetails())) {
+ errMsg = copyCmdAnswer.getDetails();
+ }
+ else {
+ errMsg = "Unable to perform host-side operation";
+ }
}
- else {
- errMsg = "Unable to perform host-side operation";
+
+ try {
+ if (errMsg == null) {
+ snapshotInfo.processEvent(Event.OperationSuccessed);
+ }
+ else {
+ snapshotInfo.processEvent(Event.OperationFailed);
+ }
+ }
+ catch (Exception ex) {
+ s_logger.debug(ex.getMessage(), ex);
}
}
- try {
- if (errMsg == null) {
- snapshotInfo.processEvent(Event.OperationSuccessed);
+ CopyCommandResult result = new CopyCommandResult(null, copyCmdAnswer);
+
+ result.setResult(errMsg);
+
+ callback.complete(result);
+ }
+ finally {
+ if (usingBackendSnapshot) {
+ deleteVolumeFromSnapshot(snapshotInfo);
+ }
+ }
+ }
+
+ /**
+ * Clones a template present on the storage to a new volume and resignatures it.
+ *
+ * @param srcData source template
+ * @param destData destination ROOT volume
+ * @param callback for async
+ */
+ private void handleCreateVolumeFromTemplateBothOnStorageSystem(DataObject srcData, DataObject destData, AsyncCompletionCallback<CopyCommandResult> callback) {
+ CopyCmdAnswer copyCmdAnswer = null;
+ String errMsg = null;
+
+ TemplateInfo templateInfo = (TemplateInfo)srcData;
+ VolumeInfo volumeInfo = (VolumeInfo)destData;
--- End diff --
What if ``srcData`` and ``destData`` are ``null``? What if they are not of the types being cast? Seems like a place to use ``Verify.verify`` with a useful error message for each of these conditions.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r59076080
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -172,78 +232,211 @@ private void validate(SnapshotInfo snapshotInfo) {
}
}
- private Void handleCreateTemplateFromSnapshot(SnapshotInfo snapshotInfo, TemplateInfo templateInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
+ private boolean usingBackendSnapshotFor(SnapshotInfo snapshotInfo) {
+ String property = getProperty(snapshotInfo.getId(), "takeSnapshot");
+
+ return Boolean.parseBoolean(property);
+ }
+
+ private void handleCreateTemplateFromSnapshot(SnapshotInfo snapshotInfo, TemplateInfo templateInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
try {
snapshotInfo.processEvent(Event.CopyingRequested);
}
catch (Exception ex) {
throw new CloudRuntimeException("This snapshot is not currently in a state where it can be used to create a template.");
}
- HostVO hostVO = getHost(snapshotInfo.getDataStore().getId());
- DataStore srcDataStore = snapshotInfo.getDataStore();
-
- String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
- int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
- CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), templateInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+ HostVO hostVO = getHost(snapshotInfo);
- String errMsg = null;
+ boolean usingBackendSnapshot = usingBackendSnapshotFor(snapshotInfo);
+ boolean computeClusterSupportsResign = computeClusterSupportsResign(hostVO.getClusterId());
- CopyCmdAnswer copyCmdAnswer = null;
+ if (usingBackendSnapshot && !computeClusterSupportsResign) {
+ throw new CloudRuntimeException("Unable to locate an applicable host with which to perform a resignature operation");
+ }
try {
- _volumeService.grantAccess(snapshotInfo, hostVO, srcDataStore);
+ if (usingBackendSnapshot) {
+ createVolumeFromSnapshot(hostVO, snapshotInfo, true);
+ }
- Map<String, String> srcDetails = getSnapshotDetails(_storagePoolDao.findById(srcDataStore.getId()), snapshotInfo);
+ DataStore srcDataStore = snapshotInfo.getDataStore();
- copyCommand.setOptions(srcDetails);
+ String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
+ int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
+ CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), templateInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+
+ String errMsg = null;
+
+ CopyCmdAnswer copyCmdAnswer = null;
- copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
- }
- catch (Exception ex) {
- throw new CloudRuntimeException(ex.getMessage());
- }
- finally {
try {
- _volumeService.revokeAccess(snapshotInfo, hostVO, srcDataStore);
+ // If we are using a back-end snapshot, then we should still have access to it from the hosts in the cluster that hostVO is in
+ // (because we passed in true as the third parameter to createVolumeFromSnapshot above).
+ if (usingBackendSnapshot == false) {
+ _volumeService.grantAccess(snapshotInfo, hostVO, srcDataStore);
+ }
+
+ Map<String, String> srcDetails = getSnapshotDetails(snapshotInfo);
+
+ copyCommand.setOptions(srcDetails);
+
+ copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
}
catch (Exception ex) {
- s_logger.debug(ex.getMessage(), ex);
+ throw new CloudRuntimeException(ex.getMessage());
}
+ finally {
+ try {
+ _volumeService.revokeAccess(snapshotInfo, hostVO, srcDataStore);
+ }
+ catch (Exception ex) {
+ s_logger.debug(ex.getMessage(), ex);
+ }
- if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
- if (copyCmdAnswer != null && copyCmdAnswer.getDetails() != null && !copyCmdAnswer.getDetails().isEmpty()) {
- errMsg = copyCmdAnswer.getDetails();
+ if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
+ if (copyCmdAnswer != null && !StringUtils.isEmpty(copyCmdAnswer.getDetails())) {
+ errMsg = copyCmdAnswer.getDetails();
+ }
+ else {
+ errMsg = "Unable to perform host-side operation";
+ }
}
- else {
- errMsg = "Unable to perform host-side operation";
+
+ try {
+ if (errMsg == null) {
+ snapshotInfo.processEvent(Event.OperationSuccessed);
+ }
+ else {
+ snapshotInfo.processEvent(Event.OperationFailed);
+ }
+ }
+ catch (Exception ex) {
+ s_logger.debug(ex.getMessage(), ex);
}
}
- try {
- if (errMsg == null) {
- snapshotInfo.processEvent(Event.OperationSuccessed);
+ CopyCommandResult result = new CopyCommandResult(null, copyCmdAnswer);
+
+ result.setResult(errMsg);
+
+ callback.complete(result);
+ }
+ finally {
+ if (usingBackendSnapshot) {
+ deleteVolumeFromSnapshot(snapshotInfo);
+ }
+ }
+ }
+
+ /**
+ * Clones a template present on the storage to a new volume and resignatures it.
+ *
+ * @param srcData source template
+ * @param destData destination ROOT volume
+ * @param callback for async
+ */
+ private void handleCreateVolumeFromTemplateBothOnStorageSystem(DataObject srcData, DataObject destData, AsyncCompletionCallback<CopyCommandResult> callback) {
+ CopyCmdAnswer copyCmdAnswer = null;
+ String errMsg = null;
+
+ TemplateInfo templateInfo = (TemplateInfo)srcData;
+ VolumeInfo volumeInfo = (VolumeInfo)destData;
+
+ HostVO hostVO = getHost(volumeInfo.getDataCenterId(), true);
+
+ if (hostVO == null) {
+ throw new CloudRuntimeException("Unable to locate a host capable of resigning in the zone with the following ID: " + volumeInfo.getDataCenterId());
+ }
+
+ boolean computeClusterSupportsResign = computeClusterSupportsResign(hostVO.getClusterId());
+
+ if (!computeClusterSupportsResign) {
+ throw new CloudRuntimeException("Unable to locate an applicable host with which to perform a resignature operation");
+ }
+
+ try {
+ VolumeDetailVO volumeDetail = new VolumeDetailVO(volumeInfo.getId(),
+ "cloneOfTemplate",
+ String.valueOf(templateInfo.getId()),
+ false);
+
+ volumeDetail = _volumeDetailsDao.persist(volumeDetail);
+
+ AsyncCallFuture<VolumeApiResult> future = _volumeService.createVolumeAsync(volumeInfo, volumeInfo.getDataStore());
+ VolumeApiResult result = future.get();
+
+ if (volumeDetail != null) {
+ _volumeDetailsDao.remove(volumeDetail.getId());
+ }
+
+ if (result.isFailed()) {
+ s_logger.debug("Failed to create a volume: " + result.getResult());
+
+ throw new CloudRuntimeException(result.getResult());
+ }
+
+ volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
+
+ volumeInfo.processEvent(Event.MigrationRequested);
+
+ volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
+
+ copyCmdAnswer = performResignature(volumeInfo, hostVO);
+
+ if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
+ if (copyCmdAnswer != null && !StringUtils.isEmpty(copyCmdAnswer.getDetails())) {
+ throw new CloudRuntimeException(copyCmdAnswer.getDetails());
}
else {
- snapshotInfo.processEvent(Event.OperationFailed);
+ throw new CloudRuntimeException("Unable to perform host-side operation");
--- End diff --
Please add some more context information to the error message to help identify the host, volume, and template involved in the operation.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by mike-tutkowski <gi...@git.apache.org>.
Github user mike-tutkowski commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r52114583
--- Diff: plugins/hypervisors/xenserver/src/com/cloud/hypervisor/xenserver/resource/CitrixResourceBase.java ---
@@ -168,7 +168,9 @@
public abstract class CitrixResourceBase implements ServerResource, HypervisorResource, VirtualRouterDeployer {
public enum SRType {
- EXT, FILE, ISCSI, ISO, LVM, LVMOHBA, LVMOISCSI, NFS;
+ // RELVMOISCSI = used for resigning metadata (like SR UUID and VDI UUID when a
+ // particular storage manager is installed on a XenServer host (for back-end snapshots to work))
+ EXT, FILE, ISCSI, ISO, LVM, LVMOHBA, LVMOISCSI, RELVMOISCSI, NFS;
--- End diff --
Done - thanks
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r59076292
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -172,78 +232,211 @@ private void validate(SnapshotInfo snapshotInfo) {
}
}
- private Void handleCreateTemplateFromSnapshot(SnapshotInfo snapshotInfo, TemplateInfo templateInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
+ private boolean usingBackendSnapshotFor(SnapshotInfo snapshotInfo) {
+ String property = getProperty(snapshotInfo.getId(), "takeSnapshot");
+
+ return Boolean.parseBoolean(property);
+ }
+
+ private void handleCreateTemplateFromSnapshot(SnapshotInfo snapshotInfo, TemplateInfo templateInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
try {
snapshotInfo.processEvent(Event.CopyingRequested);
}
catch (Exception ex) {
throw new CloudRuntimeException("This snapshot is not currently in a state where it can be used to create a template.");
}
- HostVO hostVO = getHost(snapshotInfo.getDataStore().getId());
- DataStore srcDataStore = snapshotInfo.getDataStore();
-
- String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
- int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
- CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), templateInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+ HostVO hostVO = getHost(snapshotInfo);
- String errMsg = null;
+ boolean usingBackendSnapshot = usingBackendSnapshotFor(snapshotInfo);
+ boolean computeClusterSupportsResign = computeClusterSupportsResign(hostVO.getClusterId());
- CopyCmdAnswer copyCmdAnswer = null;
+ if (usingBackendSnapshot && !computeClusterSupportsResign) {
+ throw new CloudRuntimeException("Unable to locate an applicable host with which to perform a resignature operation");
+ }
try {
- _volumeService.grantAccess(snapshotInfo, hostVO, srcDataStore);
+ if (usingBackendSnapshot) {
+ createVolumeFromSnapshot(hostVO, snapshotInfo, true);
+ }
- Map<String, String> srcDetails = getSnapshotDetails(_storagePoolDao.findById(srcDataStore.getId()), snapshotInfo);
+ DataStore srcDataStore = snapshotInfo.getDataStore();
- copyCommand.setOptions(srcDetails);
+ String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
+ int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
+ CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), templateInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+
+ String errMsg = null;
+
+ CopyCmdAnswer copyCmdAnswer = null;
- copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
- }
- catch (Exception ex) {
- throw new CloudRuntimeException(ex.getMessage());
- }
- finally {
try {
- _volumeService.revokeAccess(snapshotInfo, hostVO, srcDataStore);
+ // If we are using a back-end snapshot, then we should still have access to it from the hosts in the cluster that hostVO is in
+ // (because we passed in true as the third parameter to createVolumeFromSnapshot above).
+ if (usingBackendSnapshot == false) {
+ _volumeService.grantAccess(snapshotInfo, hostVO, srcDataStore);
+ }
+
+ Map<String, String> srcDetails = getSnapshotDetails(snapshotInfo);
+
+ copyCommand.setOptions(srcDetails);
+
+ copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
}
catch (Exception ex) {
- s_logger.debug(ex.getMessage(), ex);
+ throw new CloudRuntimeException(ex.getMessage());
}
+ finally {
+ try {
+ _volumeService.revokeAccess(snapshotInfo, hostVO, srcDataStore);
+ }
+ catch (Exception ex) {
+ s_logger.debug(ex.getMessage(), ex);
+ }
- if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
- if (copyCmdAnswer != null && copyCmdAnswer.getDetails() != null && !copyCmdAnswer.getDetails().isEmpty()) {
- errMsg = copyCmdAnswer.getDetails();
+ if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
+ if (copyCmdAnswer != null && !StringUtils.isEmpty(copyCmdAnswer.getDetails())) {
+ errMsg = copyCmdAnswer.getDetails();
+ }
+ else {
+ errMsg = "Unable to perform host-side operation";
+ }
}
- else {
- errMsg = "Unable to perform host-side operation";
+
+ try {
+ if (errMsg == null) {
+ snapshotInfo.processEvent(Event.OperationSuccessed);
+ }
+ else {
+ snapshotInfo.processEvent(Event.OperationFailed);
+ }
+ }
+ catch (Exception ex) {
+ s_logger.debug(ex.getMessage(), ex);
}
}
- try {
- if (errMsg == null) {
- snapshotInfo.processEvent(Event.OperationSuccessed);
+ CopyCommandResult result = new CopyCommandResult(null, copyCmdAnswer);
+
+ result.setResult(errMsg);
+
+ callback.complete(result);
+ }
+ finally {
+ if (usingBackendSnapshot) {
+ deleteVolumeFromSnapshot(snapshotInfo);
+ }
+ }
+ }
+
+ /**
+ * Clones a template present on the storage to a new volume and resignatures it.
+ *
+ * @param srcData source template
+ * @param destData destination ROOT volume
+ * @param callback for async
+ */
+ private void handleCreateVolumeFromTemplateBothOnStorageSystem(DataObject srcData, DataObject destData, AsyncCompletionCallback<CopyCommandResult> callback) {
+ CopyCmdAnswer copyCmdAnswer = null;
+ String errMsg = null;
+
+ TemplateInfo templateInfo = (TemplateInfo)srcData;
+ VolumeInfo volumeInfo = (VolumeInfo)destData;
+
+ HostVO hostVO = getHost(volumeInfo.getDataCenterId(), true);
+
+ if (hostVO == null) {
+ throw new CloudRuntimeException("Unable to locate a host capable of resigning in the zone with the following ID: " + volumeInfo.getDataCenterId());
+ }
+
+ boolean computeClusterSupportsResign = computeClusterSupportsResign(hostVO.getClusterId());
+
+ if (!computeClusterSupportsResign) {
+ throw new CloudRuntimeException("Unable to locate an applicable host with which to perform a resignature operation");
+ }
+
+ try {
+ VolumeDetailVO volumeDetail = new VolumeDetailVO(volumeInfo.getId(),
+ "cloneOfTemplate",
+ String.valueOf(templateInfo.getId()),
+ false);
+
+ volumeDetail = _volumeDetailsDao.persist(volumeDetail);
+
+ AsyncCallFuture<VolumeApiResult> future = _volumeService.createVolumeAsync(volumeInfo, volumeInfo.getDataStore());
+ VolumeApiResult result = future.get();
+
+ if (volumeDetail != null) {
+ _volumeDetailsDao.remove(volumeDetail.getId());
+ }
+
+ if (result.isFailed()) {
+ s_logger.debug("Failed to create a volume: " + result.getResult());
--- End diff --
Should this information be logged to ``ERROR`` rather than ``DEBUG``? Is this an error an operator would not care about?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by swill <gi...@git.apache.org>.
Github user swill commented on the pull request:
https://github.com/apache/cloudstack/pull/1403#issuecomment-213479837
## CI RESULTS
### 120/120 TESTS PASSED!
View the results below...
**Associated Uploads**
**`/tmp/MarvinLogs/DeployDataCenter__Apr_22_2016_07_25_29_9Y5PQH`:**
* [dc_entries.obj](https://objects-east.cloud.ca/v1/e465abe2f9ae4478b9fff416eab61bd9/PR1403/tmp/MarvinLogs/DeployDataCenter__Apr_22_2016_07_25_29_9Y5PQH/dc_entries.obj)
* [failed_plus_exceptions.txt](https://objects-east.cloud.ca/v1/e465abe2f9ae4478b9fff416eab61bd9/PR1403/tmp/MarvinLogs/DeployDataCenter__Apr_22_2016_07_25_29_9Y5PQH/failed_plus_exceptions.txt)
* [runinfo.txt](https://objects-east.cloud.ca/v1/e465abe2f9ae4478b9fff416eab61bd9/PR1403/tmp/MarvinLogs/DeployDataCenter__Apr_22_2016_07_25_29_9Y5PQH/runinfo.txt)
**`/tmp/MarvinLogs/test_projects_P6EJRK`:**
* [failed_plus_exceptions.txt](https://objects-east.cloud.ca/v1/e465abe2f9ae4478b9fff416eab61bd9/PR1403/tmp/MarvinLogs/test_projects_P6EJRK/failed_plus_exceptions.txt)
* [results.txt](https://objects-east.cloud.ca/v1/e465abe2f9ae4478b9fff416eab61bd9/PR1403/tmp/MarvinLogs/test_projects_P6EJRK/results.txt)
* [runinfo.txt](https://objects-east.cloud.ca/v1/e465abe2f9ae4478b9fff416eab61bd9/PR1403/tmp/MarvinLogs/test_projects_P6EJRK/runinfo.txt)
**`/tmp/MarvinLogs/test_templates_5NPAZM`:**
* [failed_plus_exceptions.txt](https://objects-east.cloud.ca/v1/e465abe2f9ae4478b9fff416eab61bd9/PR1403/tmp/MarvinLogs/test_templates_5NPAZM/failed_plus_exceptions.txt)
* [results.txt](https://objects-east.cloud.ca/v1/e465abe2f9ae4478b9fff416eab61bd9/PR1403/tmp/MarvinLogs/test_templates_5NPAZM/results.txt)
* [runinfo.txt](https://objects-east.cloud.ca/v1/e465abe2f9ae4478b9fff416eab61bd9/PR1403/tmp/MarvinLogs/test_templates_5NPAZM/runinfo.txt)
Uploads will be available until `2016-06-22 02:00:00 +0200 CEST`
*Comment created by [`upr comment`](https://github.com/swill/upr).*
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r59081613
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -361,59 +608,225 @@ private String getProperty(long snapshotId, String property) {
}
private Map<String, String> getVolumeDetails(VolumeInfo volumeInfo) {
- Map<String, String> sourceDetails = new HashMap<String, String>();
+ Map<String, String> volumeDetails = new HashMap<String, String>();
VolumeVO volumeVO = _volumeDao.findById(volumeInfo.getId());
long storagePoolId = volumeVO.getPoolId();
StoragePoolVO storagePoolVO = _storagePoolDao.findById(storagePoolId);
- sourceDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
- sourceDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
- sourceDetails.put(DiskTO.IQN, volumeVO.get_iScsiName());
+ volumeDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
+ volumeDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
+ volumeDetails.put(DiskTO.IQN, volumeVO.get_iScsiName());
ChapInfo chapInfo = _volumeService.getChapInfo(volumeInfo, volumeInfo.getDataStore());
if (chapInfo != null) {
- sourceDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, chapInfo.getInitiatorUsername());
- sourceDetails.put(DiskTO.CHAP_INITIATOR_SECRET, chapInfo.getInitiatorSecret());
- sourceDetails.put(DiskTO.CHAP_TARGET_USERNAME, chapInfo.getTargetUsername());
- sourceDetails.put(DiskTO.CHAP_TARGET_SECRET, chapInfo.getTargetSecret());
+ volumeDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, chapInfo.getInitiatorUsername());
+ volumeDetails.put(DiskTO.CHAP_INITIATOR_SECRET, chapInfo.getInitiatorSecret());
+ volumeDetails.put(DiskTO.CHAP_TARGET_USERNAME, chapInfo.getTargetUsername());
+ volumeDetails.put(DiskTO.CHAP_TARGET_SECRET, chapInfo.getTargetSecret());
+ }
+
+ return volumeDetails;
+ }
+
+ private Map<String, String> getSnapshotDetails(SnapshotInfo snapshotInfo) {
+ Map<String, String> snapshotDetails = new HashMap<String, String>();
+
+ long storagePoolId = snapshotInfo.getDataStore().getId();
+ StoragePoolVO storagePoolVO = _storagePoolDao.findById(storagePoolId);
+
+ snapshotDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
+ snapshotDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
+
+ long snapshotId = snapshotInfo.getId();
+
+ snapshotDetails.put(DiskTO.IQN, getProperty(snapshotId, DiskTO.IQN));
+
+ snapshotDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, getProperty(snapshotId, DiskTO.CHAP_INITIATOR_USERNAME));
+ snapshotDetails.put(DiskTO.CHAP_INITIATOR_SECRET, getProperty(snapshotId, DiskTO.CHAP_INITIATOR_SECRET));
+ snapshotDetails.put(DiskTO.CHAP_TARGET_USERNAME, getProperty(snapshotId, DiskTO.CHAP_TARGET_USERNAME));
+ snapshotDetails.put(DiskTO.CHAP_TARGET_SECRET, getProperty(snapshotId, DiskTO.CHAP_TARGET_SECRET));
+
+ return snapshotDetails;
+ }
+
+ private HostVO getHost(SnapshotInfo snapshotInfo) {
+ HostVO hostVO = getHost(snapshotInfo.getDataCenterId(), true);
+
+ if (hostVO == null) {
+ hostVO = getHost(snapshotInfo.getDataCenterId(), false);
+
+ if (hostVO == null) {
+ throw new CloudRuntimeException("Unable to locate an applicable host");
+ }
}
- return sourceDetails;
+ return hostVO;
}
- public HostVO getHost(long dataStoreId) {
- StoragePoolVO storagePoolVO = _storagePoolDao.findById(dataStoreId);
+ private HostVO getHost(Long zoneId, boolean computeClusterMustSupportResign) {
+ if (zoneId == null) {
+ throw new CloudRuntimeException("Zone ID cannot be null.");
+ }
- List<? extends Cluster> clusters = _mgr.searchForClusters(storagePoolVO.getDataCenterId(), new Long(0), Long.MAX_VALUE, HypervisorType.XenServer.toString());
+ List<? extends Cluster> clusters = _mgr.searchForClusters(zoneId, new Long(0), Long.MAX_VALUE, HypervisorType.XenServer.toString());
if (clusters == null) {
- throw new CloudRuntimeException("Unable to locate an applicable cluster");
+ clusters = new ArrayList<>();
}
+ Collections.shuffle(clusters, new Random(System.nanoTime()));
+
+ clusters:
for (Cluster cluster : clusters) {
if (cluster.getAllocationState() == AllocationState.Enabled) {
List<HostVO> hosts = _hostDao.findByClusterId(cluster.getId());
if (hosts != null) {
+ Collections.shuffle(hosts, new Random(System.nanoTime()));
+
for (HostVO host : hosts) {
if (host.getResourceState() == ResourceState.Enabled) {
- return host;
+ if (computeClusterMustSupportResign) {
+ if (computeClusterSupportsResign(cluster.getId())) {
+ return host;
+ }
+ else {
+ // no other host in the cluster in question should be able to satisfy our requirements here, so move on to the next cluster
+ continue clusters;
+ }
+ }
+ else {
+ return host;
+ }
}
}
}
}
}
- throw new CloudRuntimeException("Unable to locate an applicable cluster");
+ return null;
}
@Override
public void copyAsync(Map<VolumeInfo, DataStore> volumeMap, VirtualMachineTO vmTo, Host srcHost, Host destHost, AsyncCompletionCallback<CopyCommandResult> callback) {
CopyCommandResult result = new CopyCommandResult(null, null);
+
result.setResult("Unsupported operation requested for copying data.");
+
callback.complete(result);
}
+
+ private Map<String, String> getDetails(DataObject dataObj) {
+ if (dataObj instanceof VolumeInfo) {
+ return getVolumeDetails((VolumeInfo)dataObj);
+ }
+ else if (dataObj instanceof SnapshotInfo) {
+ return getSnapshotDetails((SnapshotInfo)dataObj);
+ }
+
+ throw new CloudRuntimeException("'dataObj' must be of type 'VolumeInfo' or 'SnapshotInfo'.");
+ }
+
+ private CopyCmdAnswer performResignature(DataObject dataObj, HostVO hostVO) {
+ return performResignature(dataObj, hostVO, false);
+ }
+
+ private CopyCmdAnswer performResignature(DataObject dataObj, HostVO hostVO, boolean keepGrantedAccess) {
+ long storagePoolId = dataObj.getDataStore().getId();
+ DataStore dataStore = _dataStoreMgr.getDataStore(storagePoolId, DataStoreRole.Primary);
+
+ Map<String, String> details = getDetails(dataObj);
+
+ ResignatureCommand command = new ResignatureCommand(details);
+
+ ResignatureAnswer answer = null;
+
+ try {
+ _volumeService.grantAccess(dataObj, hostVO, dataStore);
+
+ answer = (ResignatureAnswer)_agentMgr.send(hostVO.getId(), command);
+ }
+ catch (Exception ex) {
+ keepGrantedAccess = false;
+
+ throw new CloudRuntimeException(ex.getMessage());
--- End diff --
Please add contextual information to error message to express the details of the copy operation to assist with operational debugging.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r59082995
--- Diff: engine/storage/integration-test/test/org/apache/cloudstack/storage/test/FakePrimaryDataStoreDriver.java ---
@@ -44,7 +45,7 @@
boolean snapshotResult = true;
@Override
- public ChapInfo getChapInfo(VolumeInfo volumeInfo) {
+ public ChapInfo getChapInfo(DataObject dataObject) {
--- End diff --
This appears to be the default implementation from the IDE. I realize that it predates this patch, but should method do more? If not, can we strike the IDE generated comment to avoid future confusion?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r59085891
--- Diff: engine/storage/volume/src/org/apache/cloudstack/storage/datastore/PrimaryDataStoreImpl.java ---
@@ -239,10 +241,34 @@ public boolean isManaged() {
return pdsv.isManaged();
}
+ private boolean canCloneVolume() {
+ return new Boolean(getDriver().getCapabilities().get(DataStoreCapabilities.CAN_CREATE_VOLUME_FROM_VOLUME.toString()));
--- End diff --
``new Boolean`` skips the constant pool -- placing unnecessary pressure on the garbage collector. Please use ``Boolean.valueOf`` instead.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r62892410
--- Diff: core/src/com/cloud/agent/api/StartupRoutingCommand.java ---
@@ -35,7 +35,7 @@
long memory;
long dom0MinMemory;
boolean poolSync;
-
+ private boolean _supportsClonedVolumes;
--- End diff --
+1 as well. Ridding of odd Hungarian notation and "_" would be a very things for the codebase.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by mike-tutkowski <gi...@git.apache.org>.
Github user mike-tutkowski commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r63071868
--- Diff: server/src/com/cloud/storage/snapshot/SnapshotManagerImpl.java ---
@@ -144,6 +145,8 @@
@Inject
SnapshotDataStoreDao _snapshotStoreDao;
@Inject
+ private DataStoreManager _dataStoreMgr;
--- End diff --
Thanks for finding that, @DaanHoogland! I have corrected it and I will rebase against master now for @swill.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r63110910
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -132,31 +171,52 @@ public void copyAsync(DataObject srcData, DataObject destData, Host destHost, As
validate(snapshotInfo);
- boolean canHandleSrc = canHandle(srcData.getDataStore());
+ boolean canHandleSrc = canHandle(srcData);
if (canHandleSrc && destData instanceof TemplateInfo &&
(destData.getDataStore().getRole() == DataStoreRole.Image || destData.getDataStore().getRole() == DataStoreRole.ImageCache)) {
handleCreateTemplateFromSnapshot(snapshotInfo, (TemplateInfo)destData, callback);
+
return;
}
if (destData instanceof VolumeInfo) {
VolumeInfo volumeInfo = (VolumeInfo)destData;
- boolean canHandleDest = canHandle(destData.getDataStore());
+
+ boolean canHandleDest = canHandle(destData);
if (canHandleSrc && canHandleDest) {
- handleCreateVolumeFromSnapshotBothOnStorageSystem(snapshotInfo, volumeInfo, callback);
- return;
+ if (snapshotInfo.getDataStore().getId() == volumeInfo.getDataStore().getId()) {
+ handleCreateVolumeFromSnapshotBothOnStorageSystem(snapshotInfo, volumeInfo, callback);
+ return;
+ }
+ else {
+ throw new UnsupportedOperationException("This operation is not supported (DataStoreCapabilities.STORAGE_SYSTEM_SNAPSHOT " +
--- End diff --
@mike-tutkowski is it possible to add this information to log message?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r63111842
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -172,78 +258,207 @@ private void validate(SnapshotInfo snapshotInfo) {
}
}
- private Void handleCreateTemplateFromSnapshot(SnapshotInfo snapshotInfo, TemplateInfo templateInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
+ private boolean usingBackendSnapshotFor(SnapshotInfo snapshotInfo) {
+ String property = getProperty(snapshotInfo.getId(), "takeSnapshot");
+
+ return Boolean.parseBoolean(property);
+ }
+
+ private void handleCreateTemplateFromSnapshot(SnapshotInfo snapshotInfo, TemplateInfo templateInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
try {
snapshotInfo.processEvent(Event.CopyingRequested);
}
catch (Exception ex) {
throw new CloudRuntimeException("This snapshot is not currently in a state where it can be used to create a template.");
}
- HostVO hostVO = getHost(snapshotInfo.getDataStore().getId());
- DataStore srcDataStore = snapshotInfo.getDataStore();
+ HostVO hostVO = getHost(snapshotInfo);
- String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
- int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
- CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), templateInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+ boolean usingBackendSnapshot = usingBackendSnapshotFor(snapshotInfo);
+ boolean computeClusterSupportsResign = clusterDao.computeWhetherClusterSupportsResigning(hostVO.getClusterId());
- String errMsg = null;
-
- CopyCmdAnswer copyCmdAnswer = null;
+ if (usingBackendSnapshot && !computeClusterSupportsResign) {
+ throw new CloudRuntimeException("Unable to locate an applicable host with which to perform a resignature operation");
+ }
try {
- _volumeService.grantAccess(snapshotInfo, hostVO, srcDataStore);
+ if (usingBackendSnapshot) {
+ createVolumeFromSnapshot(hostVO, snapshotInfo, true);
+ }
- Map<String, String> srcDetails = getSnapshotDetails(_storagePoolDao.findById(srcDataStore.getId()), snapshotInfo);
+ DataStore srcDataStore = snapshotInfo.getDataStore();
- copyCommand.setOptions(srcDetails);
+ String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
+ int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
+ CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), templateInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+
+ String errMsg = null;
+
+ CopyCmdAnswer copyCmdAnswer = null;
- copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
- }
- catch (Exception ex) {
- throw new CloudRuntimeException(ex.getMessage());
- }
- finally {
try {
- _volumeService.revokeAccess(snapshotInfo, hostVO, srcDataStore);
+ // If we are using a back-end snapshot, then we should still have access to it from the hosts in the cluster that hostVO is in
+ // (because we passed in true as the third parameter to createVolumeFromSnapshot above).
+ if (usingBackendSnapshot == false) {
+ _volumeService.grantAccess(snapshotInfo, hostVO, srcDataStore);
+ }
+
+ Map<String, String> srcDetails = getSnapshotDetails(snapshotInfo);
+
+ copyCommand.setOptions(srcDetails);
+
+ copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
}
- catch (Exception ex) {
- s_logger.debug(ex.getMessage(), ex);
+ catch (AgentUnavailableException | OperationTimedoutException ex) {
+ throw new CloudRuntimeException("Failed to create template from snapshot : " + ex.getMessage());
}
+ finally {
+ try {
+ _volumeService.revokeAccess(snapshotInfo, hostVO, srcDataStore);
+ }
+ catch (Exception ex) {
+ s_logger.warn("error during revoking access to snapshot: " + ex.getMessage(), ex);
+ }
- if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
- if (copyCmdAnswer != null && copyCmdAnswer.getDetails() != null && !copyCmdAnswer.getDetails().isEmpty()) {
- errMsg = copyCmdAnswer.getDetails();
+ if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
+ if (copyCmdAnswer != null && !StringUtils.isEmpty(copyCmdAnswer.getDetails())) {
+ errMsg = copyCmdAnswer.getDetails();
+ }
+ else {
+ errMsg = "Unable to create template from snapshot";
+ }
}
- else {
- errMsg = "Unable to perform host-side operation";
+
+ try {
+ if (StringUtils.isEmpty(errMsg)) {
+ snapshotInfo.processEvent(Event.OperationSuccessed);
+ }
+ else {
+ snapshotInfo.processEvent(Event.OperationFailed);
+ }
+ }
+ catch (Exception ex) {
+ s_logger.warn("error while processing snapshot event" + ex.getMessage(), ex);
}
}
- try {
- if (errMsg == null) {
- snapshotInfo.processEvent(Event.OperationSuccessed);
+ CopyCommandResult result = new CopyCommandResult(null, copyCmdAnswer);
+
+ result.setResult(errMsg);
+
+ callback.complete(result);
+ }
+ finally {
+ if (usingBackendSnapshot) {
+ deleteVolumeFromSnapshot(snapshotInfo);
+ }
+ }
+ }
+
+ /**
+ * Clones a template present on the storage to a new volume and resignatures it.
+ *
+ * @param templateInfo source template
+ * @param volumeInfo destination ROOT volume
+ * @param callback for async
+ */
+ private void handleCreateVolumeFromTemplateBothOnStorageSystem(TemplateInfo templateInfo, VolumeInfo volumeInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
+ Preconditions.checkArgument(templateInfo != null, "Passing 'null' to templateInfo of handleCreateVolumeFromTemplateBothOnStorageSystem is not supported.");
+ Preconditions.checkArgument(volumeInfo != null, "Passing 'null' to volumeInfo of handleCreateVolumeFromTemplateBothOnStorageSystem is not supported.");
+
+ CopyCmdAnswer copyCmdAnswer = null;
+ String errMsg = null;
+
+ HostVO hostVO = getHost(volumeInfo.getDataCenterId(), true);
+
+ if (hostVO == null) {
+ throw new CloudRuntimeException("Unable to locate a host capable of resigning in the zone with the following ID: " + volumeInfo.getDataCenterId());
+ }
+
+ boolean computeClusterSupportsResign = clusterDao.computeWhetherClusterSupportsResigning(hostVO.getClusterId());
+
+ if (!computeClusterSupportsResign) {
+ throw new CloudRuntimeException("Unable to locate an applicable host with which to perform a resignature operation");
--- End diff --
Please include the cluster id in the error message to support operational debugging.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by mike-tutkowski <gi...@git.apache.org>.
Github user mike-tutkowski commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r63132242
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -172,78 +258,207 @@ private void validate(SnapshotInfo snapshotInfo) {
}
}
- private Void handleCreateTemplateFromSnapshot(SnapshotInfo snapshotInfo, TemplateInfo templateInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
+ private boolean usingBackendSnapshotFor(SnapshotInfo snapshotInfo) {
+ String property = getProperty(snapshotInfo.getId(), "takeSnapshot");
+
+ return Boolean.parseBoolean(property);
+ }
+
+ private void handleCreateTemplateFromSnapshot(SnapshotInfo snapshotInfo, TemplateInfo templateInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
try {
snapshotInfo.processEvent(Event.CopyingRequested);
}
catch (Exception ex) {
throw new CloudRuntimeException("This snapshot is not currently in a state where it can be used to create a template.");
}
- HostVO hostVO = getHost(snapshotInfo.getDataStore().getId());
- DataStore srcDataStore = snapshotInfo.getDataStore();
+ HostVO hostVO = getHost(snapshotInfo);
- String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
- int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
- CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), templateInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+ boolean usingBackendSnapshot = usingBackendSnapshotFor(snapshotInfo);
+ boolean computeClusterSupportsResign = clusterDao.computeWhetherClusterSupportsResigning(hostVO.getClusterId());
- String errMsg = null;
-
- CopyCmdAnswer copyCmdAnswer = null;
+ if (usingBackendSnapshot && !computeClusterSupportsResign) {
+ throw new CloudRuntimeException("Unable to locate an applicable host with which to perform a resignature operation");
+ }
try {
- _volumeService.grantAccess(snapshotInfo, hostVO, srcDataStore);
+ if (usingBackendSnapshot) {
+ createVolumeFromSnapshot(hostVO, snapshotInfo, true);
+ }
- Map<String, String> srcDetails = getSnapshotDetails(_storagePoolDao.findById(srcDataStore.getId()), snapshotInfo);
+ DataStore srcDataStore = snapshotInfo.getDataStore();
- copyCommand.setOptions(srcDetails);
+ String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
+ int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
+ CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), templateInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+
+ String errMsg = null;
+
+ CopyCmdAnswer copyCmdAnswer = null;
- copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
- }
- catch (Exception ex) {
- throw new CloudRuntimeException(ex.getMessage());
- }
- finally {
try {
- _volumeService.revokeAccess(snapshotInfo, hostVO, srcDataStore);
+ // If we are using a back-end snapshot, then we should still have access to it from the hosts in the cluster that hostVO is in
+ // (because we passed in true as the third parameter to createVolumeFromSnapshot above).
+ if (usingBackendSnapshot == false) {
+ _volumeService.grantAccess(snapshotInfo, hostVO, srcDataStore);
+ }
+
+ Map<String, String> srcDetails = getSnapshotDetails(snapshotInfo);
+
+ copyCommand.setOptions(srcDetails);
+
+ copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
}
- catch (Exception ex) {
- s_logger.debug(ex.getMessage(), ex);
+ catch (AgentUnavailableException | OperationTimedoutException ex) {
+ throw new CloudRuntimeException("Failed to create template from snapshot : " + ex.getMessage());
}
+ finally {
+ try {
+ _volumeService.revokeAccess(snapshotInfo, hostVO, srcDataStore);
+ }
+ catch (Exception ex) {
+ s_logger.warn("error during revoking access to snapshot: " + ex.getMessage(), ex);
+ }
- if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
- if (copyCmdAnswer != null && copyCmdAnswer.getDetails() != null && !copyCmdAnswer.getDetails().isEmpty()) {
- errMsg = copyCmdAnswer.getDetails();
+ if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
+ if (copyCmdAnswer != null && !StringUtils.isEmpty(copyCmdAnswer.getDetails())) {
+ errMsg = copyCmdAnswer.getDetails();
+ }
+ else {
+ errMsg = "Unable to create template from snapshot";
--- End diff --
Done
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by mike-tutkowski <gi...@git.apache.org>.
Github user mike-tutkowski commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r59110681
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -255,99 +448,153 @@ private Void handleCreateVolumeFromSnapshotBothOnStorageSystem(SnapshotInfo snap
VolumeApiResult result = future.get();
+ if (volumeDetail != null) {
+ _volumeDetailsDao.remove(volumeDetail.getId());
+ }
+
if (result.isFailed()) {
s_logger.debug("Failed to create a volume: " + result.getResult());
throw new CloudRuntimeException(result.getResult());
}
- }
- catch (Exception ex) {
- throw new CloudRuntimeException(ex.getMessage());
- }
- volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
+ volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
- volumeInfo.processEvent(Event.MigrationRequested);
+ volumeInfo.processEvent(Event.MigrationRequested);
- volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
+ volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
- HostVO hostVO = getHost(snapshotInfo.getDataStore().getId());
+ if (useCloning) {
+ copyCmdAnswer = performResignature(volumeInfo, hostVO);
+ }
+ else {
+ // asking for a XenServer host here so we don't always prefer to use XenServer hosts that support resigning
+ // even when we don't need those hosts to do this kind of copy work
+ hostVO = getHost(snapshotInfo.getDataCenterId(), false);
- String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
- int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
- CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), volumeInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+ copyCmdAnswer = performCopyOfVdi(volumeInfo, snapshotInfo, hostVO);
+ }
- CopyCmdAnswer copyCmdAnswer = null;
+ if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
+ if (copyCmdAnswer != null && !StringUtils.isEmpty(copyCmdAnswer.getDetails())) {
+ errMsg = copyCmdAnswer.getDetails();
+ }
+ else {
+ errMsg = "Unable to perform host-side operation";
+ }
+ }
+ }
+ catch (Exception ex) {
+ errMsg = ex.getMessage() != null ? ex.getMessage() : "Copy operation failed";
+ }
- try {
- _volumeService.grantAccess(snapshotInfo, hostVO, snapshotInfo.getDataStore());
- _volumeService.grantAccess(volumeInfo, hostVO, volumeInfo.getDataStore());
+ CopyCommandResult result = new CopyCommandResult(null, copyCmdAnswer);
- Map<String, String> srcDetails = getSnapshotDetails(_storagePoolDao.findById(snapshotInfo.getDataStore().getId()), snapshotInfo);
+ result.setResult(errMsg);
- copyCommand.setOptions(srcDetails);
+ callback.complete(result);
+ }
- Map<String, String> destDetails = getVolumeDetails(volumeInfo);
+ /**
+ * If the underlying storage system is making use of read-only snapshots, this gives the storage system the opportunity to
+ * create a volume from the snapshot so that we can copy the VHD file that should be inside of the snapshot to secondary storage.
+ *
+ * The resultant volume must be writable because we need to resign the SR and the VDI that should be inside of it before we copy
+ * the VHD file to secondary storage.
+ *
+ * If the storage system is using writable snapshots, then nothing need be done by that storage system here because we can just
+ * resign the SR and the VDI that should be inside of the snapshot before copying the VHD file to secondary storage.
+ */
+ private void createVolumeFromSnapshot(HostVO hostVO, SnapshotInfo snapshotInfo, boolean keepGrantedAccess) {
+ SnapshotDetailsVO snapshotDetails = handleSnapshotDetails(snapshotInfo.getId(), "tempVolume", "create");
- copyCommand.setOptions2(destDetails);
-
- copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
- }
- catch (Exception ex) {
- throw new CloudRuntimeException(ex.getMessage());
+ try {
+ snapshotInfo.getDataStore().getDriver().createAsync(snapshotInfo.getDataStore(), snapshotInfo, null);
}
finally {
- try {
- _volumeService.revokeAccess(snapshotInfo, hostVO, snapshotInfo.getDataStore());
- }
- catch (Exception ex) {
- s_logger.debug(ex.getMessage(), ex);
- }
-
- try {
- _volumeService.revokeAccess(volumeInfo, hostVO, volumeInfo.getDataStore());
- }
- catch (Exception ex) {
- s_logger.debug(ex.getMessage(), ex);
- }
+ _snapshotDetailsDao.remove(snapshotDetails.getId());
}
- String errMsg = null;
+ CopyCmdAnswer copyCmdAnswer = performResignature(snapshotInfo, hostVO, keepGrantedAccess);
if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
- if (copyCmdAnswer != null && copyCmdAnswer.getDetails() != null && !copyCmdAnswer.getDetails().isEmpty()) {
- errMsg = copyCmdAnswer.getDetails();
+ if (copyCmdAnswer != null && !StringUtils.isEmpty(copyCmdAnswer.getDetails())) {
+ throw new CloudRuntimeException(copyCmdAnswer.getDetails());
}
else {
- errMsg = "Unable to perform host-side operation";
+ throw new CloudRuntimeException("Unable to perform host-side operation");
}
}
+ }
- CopyCommandResult result = new CopyCommandResult(null, copyCmdAnswer);
+ /**
+ * If the underlying storage system needed to create a volume from a snapshot for createVolumeFromSnapshot(HostVO, SnapshotInfo), then
+ * this is its opportunity to delete that temporary volume and restore properties in snapshot_details to the way they were before the
+ * invocation of createVolumeFromSnapshot(HostVO, SnapshotInfo).
+ */
+ private void deleteVolumeFromSnapshot(SnapshotInfo snapshotInfo) {
+ SnapshotDetailsVO snapshotDetails = handleSnapshotDetails(snapshotInfo.getId(), "tempVolume", "delete");
- result.setResult(errMsg);
+ try {
+ snapshotInfo.getDataStore().getDriver().createAsync(snapshotInfo.getDataStore(), snapshotInfo, null);
+ }
+ finally {
+ _snapshotDetailsDao.remove(snapshotDetails.getId());
+ }
+ }
- callback.complete(result);
+ private SnapshotDetailsVO handleSnapshotDetails(long csSnapshotId, String name, String value) {
+ _snapshotDetailsDao.removeDetail(csSnapshotId, name);
- return null;
+ SnapshotDetailsVO snapshotDetails = new SnapshotDetailsVO(csSnapshotId, name, value, false);
+
+ return _snapshotDetailsDao.persist(snapshotDetails);
}
- private Map<String, String> getSnapshotDetails(StoragePoolVO storagePoolVO, SnapshotInfo snapshotInfo) {
- Map<String, String> details = new HashMap<String, String>();
+ private boolean canStorageSystemCreateVolumeFromVolume(SnapshotInfo snapshotInfo) {
+ boolean supportsCloningVolumeFromVolume = false;
- details.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
- details.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
+ DataStore dataStore = _dataStoreMgr.getDataStore(snapshotInfo.getDataStore().getId(), DataStoreRole.Primary);
- long snapshotId = snapshotInfo.getId();
+ Map<String, String> mapCapabilities = dataStore.getDriver().getCapabilities();
- details.put(DiskTO.IQN, getProperty(snapshotId, DiskTO.IQN));
+ if (mapCapabilities != null) {
+ String value = mapCapabilities.get(DataStoreCapabilities.CAN_CREATE_VOLUME_FROM_VOLUME.toString());
- details.put(DiskTO.CHAP_INITIATOR_USERNAME, getProperty(snapshotId, DiskTO.CHAP_INITIATOR_USERNAME));
- details.put(DiskTO.CHAP_INITIATOR_SECRET, getProperty(snapshotId, DiskTO.CHAP_INITIATOR_SECRET));
- details.put(DiskTO.CHAP_TARGET_USERNAME, getProperty(snapshotId, DiskTO.CHAP_TARGET_USERNAME));
- details.put(DiskTO.CHAP_TARGET_SECRET, getProperty(snapshotId, DiskTO.CHAP_TARGET_SECRET));
+ supportsCloningVolumeFromVolume = new Boolean(value);
+ }
- return details;
+ return supportsCloningVolumeFromVolume;
+ }
+
+ private boolean computeClusterSupportsResign(long clusterId) {
+ List<HostVO> hosts = _hostDao.findByClusterId(clusterId);
+
+ if (hosts == null) {
+ return false;
+ }
+
+ for (HostVO host : hosts) {
+ if (host == null) {
+ return false;
+ }
+
+ DetailVO hostDetail = _hostDetailsDao.findDetail(host.getId(), "supportsResign");
+
+ if (hostDetail == null) {
+ return false;
+ }
+
+ String value = hostDetail.getValue();
+
+ Boolean booleanValue = Boolean.valueOf(value);
+
+ if (booleanValue == false) {
+ return false;
+ }
+ }
--- End diff --
Are you thinking a new SQL statement like this? By the way, it looks like the editor pulls out the star that is inside of the parenthesis of my count functions.
Select count(*) = (Select count(*) From cloud.host Where cluster_id = 1)
From cloud.host_details
Where name = "supportsResign"
and value = "true"
and host_id in (Select id From cloud.host Where cluster_id = 1)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r59076763
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -172,78 +232,211 @@ private void validate(SnapshotInfo snapshotInfo) {
}
}
- private Void handleCreateTemplateFromSnapshot(SnapshotInfo snapshotInfo, TemplateInfo templateInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
+ private boolean usingBackendSnapshotFor(SnapshotInfo snapshotInfo) {
+ String property = getProperty(snapshotInfo.getId(), "takeSnapshot");
+
+ return Boolean.parseBoolean(property);
+ }
+
+ private void handleCreateTemplateFromSnapshot(SnapshotInfo snapshotInfo, TemplateInfo templateInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
try {
snapshotInfo.processEvent(Event.CopyingRequested);
}
catch (Exception ex) {
throw new CloudRuntimeException("This snapshot is not currently in a state where it can be used to create a template.");
}
- HostVO hostVO = getHost(snapshotInfo.getDataStore().getId());
- DataStore srcDataStore = snapshotInfo.getDataStore();
-
- String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
- int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
- CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), templateInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+ HostVO hostVO = getHost(snapshotInfo);
- String errMsg = null;
+ boolean usingBackendSnapshot = usingBackendSnapshotFor(snapshotInfo);
+ boolean computeClusterSupportsResign = computeClusterSupportsResign(hostVO.getClusterId());
- CopyCmdAnswer copyCmdAnswer = null;
+ if (usingBackendSnapshot && !computeClusterSupportsResign) {
+ throw new CloudRuntimeException("Unable to locate an applicable host with which to perform a resignature operation");
+ }
try {
- _volumeService.grantAccess(snapshotInfo, hostVO, srcDataStore);
+ if (usingBackendSnapshot) {
+ createVolumeFromSnapshot(hostVO, snapshotInfo, true);
+ }
- Map<String, String> srcDetails = getSnapshotDetails(_storagePoolDao.findById(srcDataStore.getId()), snapshotInfo);
+ DataStore srcDataStore = snapshotInfo.getDataStore();
- copyCommand.setOptions(srcDetails);
+ String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
+ int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
+ CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), templateInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+
+ String errMsg = null;
+
+ CopyCmdAnswer copyCmdAnswer = null;
- copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
- }
- catch (Exception ex) {
- throw new CloudRuntimeException(ex.getMessage());
- }
- finally {
try {
- _volumeService.revokeAccess(snapshotInfo, hostVO, srcDataStore);
+ // If we are using a back-end snapshot, then we should still have access to it from the hosts in the cluster that hostVO is in
+ // (because we passed in true as the third parameter to createVolumeFromSnapshot above).
+ if (usingBackendSnapshot == false) {
+ _volumeService.grantAccess(snapshotInfo, hostVO, srcDataStore);
+ }
+
+ Map<String, String> srcDetails = getSnapshotDetails(snapshotInfo);
+
+ copyCommand.setOptions(srcDetails);
+
+ copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
}
catch (Exception ex) {
- s_logger.debug(ex.getMessage(), ex);
+ throw new CloudRuntimeException(ex.getMessage());
}
+ finally {
+ try {
+ _volumeService.revokeAccess(snapshotInfo, hostVO, srcDataStore);
+ }
+ catch (Exception ex) {
+ s_logger.debug(ex.getMessage(), ex);
+ }
- if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
- if (copyCmdAnswer != null && copyCmdAnswer.getDetails() != null && !copyCmdAnswer.getDetails().isEmpty()) {
- errMsg = copyCmdAnswer.getDetails();
+ if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
+ if (copyCmdAnswer != null && !StringUtils.isEmpty(copyCmdAnswer.getDetails())) {
+ errMsg = copyCmdAnswer.getDetails();
+ }
+ else {
+ errMsg = "Unable to perform host-side operation";
+ }
}
- else {
- errMsg = "Unable to perform host-side operation";
+
+ try {
+ if (errMsg == null) {
+ snapshotInfo.processEvent(Event.OperationSuccessed);
+ }
+ else {
+ snapshotInfo.processEvent(Event.OperationFailed);
+ }
+ }
+ catch (Exception ex) {
+ s_logger.debug(ex.getMessage(), ex);
}
}
- try {
- if (errMsg == null) {
- snapshotInfo.processEvent(Event.OperationSuccessed);
+ CopyCommandResult result = new CopyCommandResult(null, copyCmdAnswer);
+
+ result.setResult(errMsg);
+
+ callback.complete(result);
+ }
+ finally {
+ if (usingBackendSnapshot) {
+ deleteVolumeFromSnapshot(snapshotInfo);
+ }
+ }
+ }
+
+ /**
+ * Clones a template present on the storage to a new volume and resignatures it.
+ *
+ * @param srcData source template
+ * @param destData destination ROOT volume
+ * @param callback for async
+ */
+ private void handleCreateVolumeFromTemplateBothOnStorageSystem(DataObject srcData, DataObject destData, AsyncCompletionCallback<CopyCommandResult> callback) {
+ CopyCmdAnswer copyCmdAnswer = null;
+ String errMsg = null;
+
+ TemplateInfo templateInfo = (TemplateInfo)srcData;
+ VolumeInfo volumeInfo = (VolumeInfo)destData;
+
+ HostVO hostVO = getHost(volumeInfo.getDataCenterId(), true);
+
+ if (hostVO == null) {
+ throw new CloudRuntimeException("Unable to locate a host capable of resigning in the zone with the following ID: " + volumeInfo.getDataCenterId());
+ }
+
+ boolean computeClusterSupportsResign = computeClusterSupportsResign(hostVO.getClusterId());
+
+ if (!computeClusterSupportsResign) {
+ throw new CloudRuntimeException("Unable to locate an applicable host with which to perform a resignature operation");
+ }
+
+ try {
+ VolumeDetailVO volumeDetail = new VolumeDetailVO(volumeInfo.getId(),
+ "cloneOfTemplate",
+ String.valueOf(templateInfo.getId()),
+ false);
+
+ volumeDetail = _volumeDetailsDao.persist(volumeDetail);
+
+ AsyncCallFuture<VolumeApiResult> future = _volumeService.createVolumeAsync(volumeInfo, volumeInfo.getDataStore());
+ VolumeApiResult result = future.get();
+
+ if (volumeDetail != null) {
+ _volumeDetailsDao.remove(volumeDetail.getId());
+ }
+
+ if (result.isFailed()) {
+ s_logger.debug("Failed to create a volume: " + result.getResult());
+
+ throw new CloudRuntimeException(result.getResult());
+ }
+
+ volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
+
+ volumeInfo.processEvent(Event.MigrationRequested);
+
+ volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
+
+ copyCmdAnswer = performResignature(volumeInfo, hostVO);
+
+ if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
+ if (copyCmdAnswer != null && !StringUtils.isEmpty(copyCmdAnswer.getDetails())) {
+ throw new CloudRuntimeException(copyCmdAnswer.getDetails());
}
else {
- snapshotInfo.processEvent(Event.OperationFailed);
+ throw new CloudRuntimeException("Unable to perform host-side operation");
}
}
- catch (Exception ex) {
- s_logger.debug(ex.getMessage(), ex);
+ } catch (Exception ex) {
+ try {
+ volumeInfo.getDataStore().getDriver().deleteAsync(volumeInfo.getDataStore(), volumeInfo, null);
+ } catch (Exception e) {
--- End diff --
Why are all checked and unchecked exceptions being caught?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by swill <gi...@git.apache.org>.
Github user swill commented on the pull request:
https://github.com/apache/cloudstack/pull/1403#issuecomment-210009483
This does not compile:
```
[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR] /data/git/cs1/cloudstack/dist/rpmbuild/BUILD/cloudstack-4.9.0-SNAPSHOT/plugins/api/solidfire-intg-test/src/org/apache/cloudstack/solidfire/ApiSolidFireServiceImpl.java:[93,68] error: cannot find symbol
[INFO] 1 error
```
It fails here:
```
[INFO] Apache CloudStack Plugin - Storage Volume default provider SUCCESS [2.494s]
[INFO] Apache CloudStack Plugin - Storage Volume SolidFire Provider SUCCESS [6.477s]
[INFO] Apache CloudStack Plugin - API SolidFire .......... FAILURE [2.601s]
[INFO] Apache CloudStack Plugin - API Discovery .......... SKIPPED
[INFO] Apache CloudStack Plugin - ACL Static Role Based .. SKIPPED
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by GabrielBrascher <gi...@git.apache.org>.
Github user GabrielBrascher commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r52108693
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -255,99 +332,149 @@ private Void handleCreateVolumeFromSnapshotBothOnStorageSystem(SnapshotInfo snap
VolumeApiResult result = future.get();
+ if (volumeDetail != null) {
+ _volumeDetailsDao.remove(volumeDetail.getId());
+ }
+
if (result.isFailed()) {
s_logger.debug("Failed to create a volume: " + result.getResult());
throw new CloudRuntimeException(result.getResult());
}
- }
- catch (Exception ex) {
- throw new CloudRuntimeException(ex.getMessage());
- }
-
- volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
- volumeInfo.processEvent(Event.MigrationRequested);
+ volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
- volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
+ volumeInfo.processEvent(Event.MigrationRequested);
- HostVO hostVO = getHost(snapshotInfo.getDataStore().getId());
+ volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
- String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
- int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
- CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), volumeInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+ if (useCloning) {
+ copyCmdAnswer = performResignature(volumeInfo, hostVO);
+ }
+ else {
+ // asking for a XenServer host here so we don't always prefer to use XenServer hosts that support resigning
+ // even when we don't need those hosts to do this kind of copy work
+ hostVO = getXenServerHost(snapshotInfo.getDataCenterId(), false);
- CopyCmdAnswer copyCmdAnswer = null;
+ copyCmdAnswer = performCopyOfVdi(volumeInfo, snapshotInfo, hostVO);
+ }
- try {
- _volumeService.grantAccess(snapshotInfo, hostVO, snapshotInfo.getDataStore());
- _volumeService.grantAccess(volumeInfo, hostVO, volumeInfo.getDataStore());
+ if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
+ if (copyCmdAnswer != null && copyCmdAnswer.getDetails() != null && !copyCmdAnswer.getDetails().isEmpty()) {
+ errMsg = copyCmdAnswer.getDetails();
+ }
+ else {
+ errMsg = "Unable to perform host-side operation";
+ }
+ }
+ }
+ catch (Exception ex) {
+ errMsg = ex.getMessage() != null ? ex.getMessage() : "Copy operation failed";
+ }
- Map<String, String> srcDetails = getSnapshotDetails(_storagePoolDao.findById(snapshotInfo.getDataStore().getId()), snapshotInfo);
+ CopyCommandResult result = new CopyCommandResult(null, copyCmdAnswer);
- copyCommand.setOptions(srcDetails);
+ result.setResult(errMsg);
- Map<String, String> destDetails = getVolumeDetails(volumeInfo);
+ callback.complete(result);
+ }
- copyCommand.setOptions2(destDetails);
+ // If the underlying storage system is making use of read-only snapshots, this gives the storage system the opportunity to
+ // create a volume from the snapshot so that we can copy the VHD file that should be inside of the snapshot to secondary storage.
+ //
+ // The resultant volume must be writable because we need to resign the SR and the VDI that should be inside of it before we copy
+ // the VHD file to secondary storage.
+ //
+ // If the storage system is using writable snapshots, then nothing need be done by that storage system here because we can just
+ // resign the SR and the VDI that should be inside of the snapshot before copying the VHD file to secondary storage.
+ private void createVolumeFromSnapshot(HostVO hostVO, SnapshotInfo snapshotInfo, boolean keepGrantedAccess) {
--- End diff --
@mike-tutkowski Could you please use Javadoc (`/** <documentation> */`) instead of comments (`// <documentation>`)?
As you already wrote a good documentation, I think it would enrich your work.
Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r59082041
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -361,59 +608,225 @@ private String getProperty(long snapshotId, String property) {
}
private Map<String, String> getVolumeDetails(VolumeInfo volumeInfo) {
- Map<String, String> sourceDetails = new HashMap<String, String>();
+ Map<String, String> volumeDetails = new HashMap<String, String>();
VolumeVO volumeVO = _volumeDao.findById(volumeInfo.getId());
long storagePoolId = volumeVO.getPoolId();
StoragePoolVO storagePoolVO = _storagePoolDao.findById(storagePoolId);
- sourceDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
- sourceDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
- sourceDetails.put(DiskTO.IQN, volumeVO.get_iScsiName());
+ volumeDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
+ volumeDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
+ volumeDetails.put(DiskTO.IQN, volumeVO.get_iScsiName());
ChapInfo chapInfo = _volumeService.getChapInfo(volumeInfo, volumeInfo.getDataStore());
if (chapInfo != null) {
- sourceDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, chapInfo.getInitiatorUsername());
- sourceDetails.put(DiskTO.CHAP_INITIATOR_SECRET, chapInfo.getInitiatorSecret());
- sourceDetails.put(DiskTO.CHAP_TARGET_USERNAME, chapInfo.getTargetUsername());
- sourceDetails.put(DiskTO.CHAP_TARGET_SECRET, chapInfo.getTargetSecret());
+ volumeDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, chapInfo.getInitiatorUsername());
+ volumeDetails.put(DiskTO.CHAP_INITIATOR_SECRET, chapInfo.getInitiatorSecret());
+ volumeDetails.put(DiskTO.CHAP_TARGET_USERNAME, chapInfo.getTargetUsername());
+ volumeDetails.put(DiskTO.CHAP_TARGET_SECRET, chapInfo.getTargetSecret());
+ }
+
+ return volumeDetails;
+ }
+
+ private Map<String, String> getSnapshotDetails(SnapshotInfo snapshotInfo) {
+ Map<String, String> snapshotDetails = new HashMap<String, String>();
+
+ long storagePoolId = snapshotInfo.getDataStore().getId();
+ StoragePoolVO storagePoolVO = _storagePoolDao.findById(storagePoolId);
+
+ snapshotDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
+ snapshotDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
+
+ long snapshotId = snapshotInfo.getId();
+
+ snapshotDetails.put(DiskTO.IQN, getProperty(snapshotId, DiskTO.IQN));
+
+ snapshotDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, getProperty(snapshotId, DiskTO.CHAP_INITIATOR_USERNAME));
+ snapshotDetails.put(DiskTO.CHAP_INITIATOR_SECRET, getProperty(snapshotId, DiskTO.CHAP_INITIATOR_SECRET));
+ snapshotDetails.put(DiskTO.CHAP_TARGET_USERNAME, getProperty(snapshotId, DiskTO.CHAP_TARGET_USERNAME));
+ snapshotDetails.put(DiskTO.CHAP_TARGET_SECRET, getProperty(snapshotId, DiskTO.CHAP_TARGET_SECRET));
+
+ return snapshotDetails;
+ }
+
+ private HostVO getHost(SnapshotInfo snapshotInfo) {
+ HostVO hostVO = getHost(snapshotInfo.getDataCenterId(), true);
+
+ if (hostVO == null) {
+ hostVO = getHost(snapshotInfo.getDataCenterId(), false);
+
+ if (hostVO == null) {
+ throw new CloudRuntimeException("Unable to locate an applicable host");
+ }
}
- return sourceDetails;
+ return hostVO;
}
- public HostVO getHost(long dataStoreId) {
- StoragePoolVO storagePoolVO = _storagePoolDao.findById(dataStoreId);
+ private HostVO getHost(Long zoneId, boolean computeClusterMustSupportResign) {
+ if (zoneId == null) {
+ throw new CloudRuntimeException("Zone ID cannot be null.");
+ }
- List<? extends Cluster> clusters = _mgr.searchForClusters(storagePoolVO.getDataCenterId(), new Long(0), Long.MAX_VALUE, HypervisorType.XenServer.toString());
+ List<? extends Cluster> clusters = _mgr.searchForClusters(zoneId, new Long(0), Long.MAX_VALUE, HypervisorType.XenServer.toString());
if (clusters == null) {
- throw new CloudRuntimeException("Unable to locate an applicable cluster");
+ clusters = new ArrayList<>();
}
+ Collections.shuffle(clusters, new Random(System.nanoTime()));
+
+ clusters:
for (Cluster cluster : clusters) {
if (cluster.getAllocationState() == AllocationState.Enabled) {
List<HostVO> hosts = _hostDao.findByClusterId(cluster.getId());
if (hosts != null) {
+ Collections.shuffle(hosts, new Random(System.nanoTime()));
+
for (HostVO host : hosts) {
if (host.getResourceState() == ResourceState.Enabled) {
- return host;
+ if (computeClusterMustSupportResign) {
+ if (computeClusterSupportsResign(cluster.getId())) {
+ return host;
+ }
+ else {
+ // no other host in the cluster in question should be able to satisfy our requirements here, so move on to the next cluster
+ continue clusters;
+ }
+ }
+ else {
+ return host;
+ }
}
}
}
}
}
- throw new CloudRuntimeException("Unable to locate an applicable cluster");
+ return null;
}
@Override
public void copyAsync(Map<VolumeInfo, DataStore> volumeMap, VirtualMachineTO vmTo, Host srcHost, Host destHost, AsyncCompletionCallback<CopyCommandResult> callback) {
CopyCommandResult result = new CopyCommandResult(null, null);
+
result.setResult("Unsupported operation requested for copying data.");
+
callback.complete(result);
}
+
+ private Map<String, String> getDetails(DataObject dataObj) {
+ if (dataObj instanceof VolumeInfo) {
+ return getVolumeDetails((VolumeInfo)dataObj);
+ }
+ else if (dataObj instanceof SnapshotInfo) {
+ return getSnapshotDetails((SnapshotInfo)dataObj);
+ }
+
+ throw new CloudRuntimeException("'dataObj' must be of type 'VolumeInfo' or 'SnapshotInfo'.");
+ }
+
+ private CopyCmdAnswer performResignature(DataObject dataObj, HostVO hostVO) {
+ return performResignature(dataObj, hostVO, false);
+ }
+
+ private CopyCmdAnswer performResignature(DataObject dataObj, HostVO hostVO, boolean keepGrantedAccess) {
+ long storagePoolId = dataObj.getDataStore().getId();
+ DataStore dataStore = _dataStoreMgr.getDataStore(storagePoolId, DataStoreRole.Primary);
+
+ Map<String, String> details = getDetails(dataObj);
+
+ ResignatureCommand command = new ResignatureCommand(details);
+
+ ResignatureAnswer answer = null;
+
+ try {
+ _volumeService.grantAccess(dataObj, hostVO, dataStore);
+
+ answer = (ResignatureAnswer)_agentMgr.send(hostVO.getId(), command);
+ }
+ catch (Exception ex) {
+ keepGrantedAccess = false;
+
+ throw new CloudRuntimeException(ex.getMessage());
+ }
+ finally {
+ if (keepGrantedAccess == false) {
+ try {
+ _volumeService.revokeAccess(dataObj, hostVO, dataStore);
+ }
+ catch (Exception ex) {
+ s_logger.debug(ex.getMessage(), ex);
+ }
+ }
+ }
+
+ if (answer == null || !answer.getResult()) {
+ final String errMsg;
+
+ if (answer != null && answer.getDetails() != null && !answer.getDetails().isEmpty()) {
+ errMsg = answer.getDetails();
+ }
+ else {
+ errMsg = "Unable to perform resignature operation";
+ }
+
+ throw new CloudRuntimeException(errMsg);
+ }
+
+ VolumeObjectTO newVolume = new VolumeObjectTO();
+
+ newVolume.setSize(answer.getSize());
+ newVolume.setPath(answer.getPath());
+ newVolume.setFormat(answer.getFormat());
+
+ return new CopyCmdAnswer(newVolume);
+ }
+
+ private CopyCmdAnswer performCopyOfVdi(VolumeInfo volumeInfo, SnapshotInfo snapshotInfo, HostVO hostVO) {
+ String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
+ int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
+ CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), volumeInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+
+ CopyCmdAnswer copyCmdAnswer = null;
+
+ try {
+ _volumeService.grantAccess(snapshotInfo, hostVO, snapshotInfo.getDataStore());
+ _volumeService.grantAccess(volumeInfo, hostVO, volumeInfo.getDataStore());
+
+ Map<String, String> srcDetails = getSnapshotDetails(snapshotInfo);
+
+ copyCommand.setOptions(srcDetails);
+
+ Map<String, String> destDetails = getVolumeDetails(volumeInfo);
+
+ copyCommand.setOptions2(destDetails);
+
+ copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
+ }
+ catch (Exception ex) {
+ throw new CloudRuntimeException(ex.getMessage());
--- End diff --
Please add contextual information about the copy VDI operation to error message to assist with operational debugging.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by DaanHoogland <gi...@git.apache.org>.
Github user DaanHoogland commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r62867309
--- Diff: engine/api/src/org/apache/cloudstack/engine/subsystem/api/storage/PrimaryDataStoreInfo.java ---
@@ -36,6 +36,7 @@
static final String CHAP_INITIATOR_SECRET = "chapInitiatorSecret";
static final String CHAP_TARGET_USERNAME = "chapTargetUsername";
static final String CHAP_TARGET_SECRET = "chapTargetSecret";
+ static final String REMOVE_AFTER_COPY = "removeAfterCopy";
--- End diff --
not lethal but these strings seem to warrant an enum as well, do they?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r63111572
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -172,78 +258,207 @@ private void validate(SnapshotInfo snapshotInfo) {
}
}
- private Void handleCreateTemplateFromSnapshot(SnapshotInfo snapshotInfo, TemplateInfo templateInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
+ private boolean usingBackendSnapshotFor(SnapshotInfo snapshotInfo) {
+ String property = getProperty(snapshotInfo.getId(), "takeSnapshot");
+
+ return Boolean.parseBoolean(property);
+ }
+
+ private void handleCreateTemplateFromSnapshot(SnapshotInfo snapshotInfo, TemplateInfo templateInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
try {
snapshotInfo.processEvent(Event.CopyingRequested);
}
catch (Exception ex) {
throw new CloudRuntimeException("This snapshot is not currently in a state where it can be used to create a template.");
}
- HostVO hostVO = getHost(snapshotInfo.getDataStore().getId());
- DataStore srcDataStore = snapshotInfo.getDataStore();
+ HostVO hostVO = getHost(snapshotInfo);
- String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
- int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
- CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), templateInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+ boolean usingBackendSnapshot = usingBackendSnapshotFor(snapshotInfo);
+ boolean computeClusterSupportsResign = clusterDao.computeWhetherClusterSupportsResigning(hostVO.getClusterId());
- String errMsg = null;
-
- CopyCmdAnswer copyCmdAnswer = null;
+ if (usingBackendSnapshot && !computeClusterSupportsResign) {
+ throw new CloudRuntimeException("Unable to locate an applicable host with which to perform a resignature operation");
--- End diff --
Please include the cluster id in the error message to assist operational debugging.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by mike-tutkowski <gi...@git.apache.org>.
Github user mike-tutkowski commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r63131753
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -361,59 +608,225 @@ private String getProperty(long snapshotId, String property) {
}
private Map<String, String> getVolumeDetails(VolumeInfo volumeInfo) {
- Map<String, String> sourceDetails = new HashMap<String, String>();
+ Map<String, String> volumeDetails = new HashMap<String, String>();
VolumeVO volumeVO = _volumeDao.findById(volumeInfo.getId());
long storagePoolId = volumeVO.getPoolId();
StoragePoolVO storagePoolVO = _storagePoolDao.findById(storagePoolId);
- sourceDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
- sourceDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
- sourceDetails.put(DiskTO.IQN, volumeVO.get_iScsiName());
+ volumeDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
+ volumeDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
+ volumeDetails.put(DiskTO.IQN, volumeVO.get_iScsiName());
ChapInfo chapInfo = _volumeService.getChapInfo(volumeInfo, volumeInfo.getDataStore());
if (chapInfo != null) {
- sourceDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, chapInfo.getInitiatorUsername());
- sourceDetails.put(DiskTO.CHAP_INITIATOR_SECRET, chapInfo.getInitiatorSecret());
- sourceDetails.put(DiskTO.CHAP_TARGET_USERNAME, chapInfo.getTargetUsername());
- sourceDetails.put(DiskTO.CHAP_TARGET_SECRET, chapInfo.getTargetSecret());
+ volumeDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, chapInfo.getInitiatorUsername());
+ volumeDetails.put(DiskTO.CHAP_INITIATOR_SECRET, chapInfo.getInitiatorSecret());
+ volumeDetails.put(DiskTO.CHAP_TARGET_USERNAME, chapInfo.getTargetUsername());
+ volumeDetails.put(DiskTO.CHAP_TARGET_SECRET, chapInfo.getTargetSecret());
+ }
+
+ return volumeDetails;
+ }
+
+ private Map<String, String> getSnapshotDetails(SnapshotInfo snapshotInfo) {
+ Map<String, String> snapshotDetails = new HashMap<String, String>();
+
+ long storagePoolId = snapshotInfo.getDataStore().getId();
+ StoragePoolVO storagePoolVO = _storagePoolDao.findById(storagePoolId);
+
+ snapshotDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
+ snapshotDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
+
+ long snapshotId = snapshotInfo.getId();
+
+ snapshotDetails.put(DiskTO.IQN, getProperty(snapshotId, DiskTO.IQN));
+
+ snapshotDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, getProperty(snapshotId, DiskTO.CHAP_INITIATOR_USERNAME));
+ snapshotDetails.put(DiskTO.CHAP_INITIATOR_SECRET, getProperty(snapshotId, DiskTO.CHAP_INITIATOR_SECRET));
+ snapshotDetails.put(DiskTO.CHAP_TARGET_USERNAME, getProperty(snapshotId, DiskTO.CHAP_TARGET_USERNAME));
+ snapshotDetails.put(DiskTO.CHAP_TARGET_SECRET, getProperty(snapshotId, DiskTO.CHAP_TARGET_SECRET));
+
+ return snapshotDetails;
+ }
+
+ private HostVO getHost(SnapshotInfo snapshotInfo) {
+ HostVO hostVO = getHost(snapshotInfo.getDataCenterId(), true);
+
+ if (hostVO == null) {
+ hostVO = getHost(snapshotInfo.getDataCenterId(), false);
+
+ if (hostVO == null) {
+ throw new CloudRuntimeException("Unable to locate an applicable host");
+ }
}
- return sourceDetails;
+ return hostVO;
}
- public HostVO getHost(long dataStoreId) {
- StoragePoolVO storagePoolVO = _storagePoolDao.findById(dataStoreId);
+ private HostVO getHost(Long zoneId, boolean computeClusterMustSupportResign) {
+ if (zoneId == null) {
+ throw new CloudRuntimeException("Zone ID cannot be null.");
+ }
- List<? extends Cluster> clusters = _mgr.searchForClusters(storagePoolVO.getDataCenterId(), new Long(0), Long.MAX_VALUE, HypervisorType.XenServer.toString());
+ List<? extends Cluster> clusters = _mgr.searchForClusters(zoneId, new Long(0), Long.MAX_VALUE, HypervisorType.XenServer.toString());
if (clusters == null) {
- throw new CloudRuntimeException("Unable to locate an applicable cluster");
+ clusters = new ArrayList<>();
}
+ Collections.shuffle(clusters, new Random(System.nanoTime()));
+
+ clusters:
for (Cluster cluster : clusters) {
if (cluster.getAllocationState() == AllocationState.Enabled) {
List<HostVO> hosts = _hostDao.findByClusterId(cluster.getId());
if (hosts != null) {
+ Collections.shuffle(hosts, new Random(System.nanoTime()));
+
for (HostVO host : hosts) {
if (host.getResourceState() == ResourceState.Enabled) {
- return host;
+ if (computeClusterMustSupportResign) {
+ if (computeClusterSupportsResign(cluster.getId())) {
+ return host;
+ }
+ else {
+ // no other host in the cluster in question should be able to satisfy our requirements here, so move on to the next cluster
+ continue clusters;
+ }
+ }
+ else {
+ return host;
+ }
}
}
}
}
}
- throw new CloudRuntimeException("Unable to locate an applicable cluster");
+ return null;
}
@Override
public void copyAsync(Map<VolumeInfo, DataStore> volumeMap, VirtualMachineTO vmTo, Host srcHost, Host destHost, AsyncCompletionCallback<CopyCommandResult> callback) {
CopyCommandResult result = new CopyCommandResult(null, null);
+
result.setResult("Unsupported operation requested for copying data.");
+
callback.complete(result);
}
+
+ private Map<String, String> getDetails(DataObject dataObj) {
+ if (dataObj instanceof VolumeInfo) {
+ return getVolumeDetails((VolumeInfo)dataObj);
+ }
+ else if (dataObj instanceof SnapshotInfo) {
+ return getSnapshotDetails((SnapshotInfo)dataObj);
+ }
+
+ throw new CloudRuntimeException("'dataObj' must be of type 'VolumeInfo' or 'SnapshotInfo'.");
+ }
+
+ private CopyCmdAnswer performResignature(DataObject dataObj, HostVO hostVO) {
+ return performResignature(dataObj, hostVO, false);
+ }
+
+ private CopyCmdAnswer performResignature(DataObject dataObj, HostVO hostVO, boolean keepGrantedAccess) {
+ long storagePoolId = dataObj.getDataStore().getId();
+ DataStore dataStore = _dataStoreMgr.getDataStore(storagePoolId, DataStoreRole.Primary);
+
+ Map<String, String> details = getDetails(dataObj);
+
+ ResignatureCommand command = new ResignatureCommand(details);
+
+ ResignatureAnswer answer = null;
+
+ try {
+ _volumeService.grantAccess(dataObj, hostVO, dataStore);
+
+ answer = (ResignatureAnswer)_agentMgr.send(hostVO.getId(), command);
+ }
+ catch (Exception ex) {
+ keepGrantedAccess = false;
+
+ throw new CloudRuntimeException(ex.getMessage());
+ }
+ finally {
+ if (keepGrantedAccess == false) {
+ try {
+ _volumeService.revokeAccess(dataObj, hostVO, dataStore);
+ }
+ catch (Exception ex) {
+ s_logger.debug(ex.getMessage(), ex);
+ }
+ }
+ }
+
+ if (answer == null || !answer.getResult()) {
+ final String errMsg;
+
+ if (answer != null && answer.getDetails() != null && !answer.getDetails().isEmpty()) {
+ errMsg = answer.getDetails();
+ }
+ else {
+ errMsg = "Unable to perform resignature operation";
--- End diff --
I added the name of the class and method, but line 775 ("Unable to perform resignature operation") should really not end up being executed unless a bad details message (null or empty string) is passed back from the host.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by mike-tutkowski <gi...@git.apache.org>.
Github user mike-tutkowski commented on the pull request:
https://github.com/apache/cloudstack/pull/1403#issuecomment-218869277
@DaanHoogland I agree we have some strange patterns in the codebase. For this feature, I simply followed those patterns when I saw them because I wasn't sure why they were coded that way (maybe for AOP?) and was afraid to deviate from it unless I better understood why it was the way it was in the first place (for example, some "Void" return types instead of "void" return types).
I did make the changes you suggested, though, Daan. Hopefully there wasn't any good reason in the first place why some of those Void return types were in there.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r59074290
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -114,6 +142,17 @@ private boolean canHandle(DataStore dataStore) {
return true;
}
+ } else if (dataObject instanceof TemplateInfo) {
+ // If the storage system can clone volumes, we can cache templates on it.
+ String value = mapCapabilities.get(DataStoreCapabilities.CAN_CREATE_VOLUME_FROM_VOLUME.toString());
--- End diff --
Is it possible for ``value`` to be ``null`` or blank?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r59075640
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -80,32 +92,48 @@
@Inject private AgentManager _agentMgr;
@Inject private ConfigurationDao _configDao;
+ @Inject private DataStoreManager _dataStoreMgr;
@Inject private DiskOfferingDao _diskOfferingDao;
@Inject private HostDao _hostDao;
+ @Inject private HostDetailsDao _hostDetailsDao;
@Inject private ManagementService _mgr;
@Inject private PrimaryDataStoreDao _storagePoolDao;
@Inject private SnapshotDao _snapshotDao;
@Inject private SnapshotDetailsDao _snapshotDetailsDao;
@Inject private VolumeDao _volumeDao;
@Inject private VolumeDataFactory _volumeDataFactory;
+ @Inject private VolumeDetailsDao _volumeDetailsDao;
@Inject private VolumeService _volumeService;
@Override
public StrategyPriority canHandle(DataObject srcData, DataObject destData) {
if (srcData instanceof SnapshotInfo) {
- if (canHandle(srcData.getDataStore()) || canHandle(destData.getDataStore())) {
+ if (canHandle(srcData) || canHandle(destData)) {
return StrategyPriority.HIGHEST;
}
}
+ if (srcData instanceof TemplateInfo && destData instanceof VolumeInfo &&
+ (srcData.getDataStore().getId() == destData.getDataStore().getId()) &&
+ (canHandle(srcData) || canHandle(destData))) {
+ // Both source and dest are on the same storage, so just clone them.
+ return StrategyPriority.HIGHEST;
+ }
+
return StrategyPriority.CANT_HANDLE;
}
- private boolean canHandle(DataStore dataStore) {
+ private boolean canHandle(DataObject dataObject) {
--- End diff --
@mike-tutkowski while it's unlikely, we should defined against if for nothing more than providing a better error than an NPE. Seems like a good place to use ``Preconditions.checkArgument`` with a meaningful error message. It also better pinpoints the origin of the failure.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by mike-tutkowski <gi...@git.apache.org>.
Github user mike-tutkowski commented on the pull request:
https://github.com/apache/cloudstack/pull/1403#issuecomment-210081474
@swill I just pushed a new commit to rename the SolidFire integration testing project (which you saw as failing).
While that should not have anything to do with a failure, I'd be curious to have you re-run your build because ApiSolidFireServiceImpl (which was failing) should not even be a part of the codebase in that SHA.
Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by mike-tutkowski <gi...@git.apache.org>.
Github user mike-tutkowski commented on the pull request:
https://github.com/apache/cloudstack/pull/1403#issuecomment-207710030
@jburwell Thanks for the comments. I pushed a new commit that implements most of your comments. I'm still looking at one or two of them.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on the pull request:
https://github.com/apache/cloudstack/pull/1403#issuecomment-208417118
@mike-tutkowski ``null`` values in a ``Map`` are extremely ugly. It seems like filtering them at the time a ``Map`` is created would make things simpler overall (i.e. no need to check that every value retrieved is not ``null``. Using Guava, you can filter a map cleanly as follows:
```
ImmutableMap<String, Object> copy = ImmutableMap.of(Maps.filterValues(map, new Predicate<Object>() {
apply(Object input) {
return input != null;
}
}));
```
This type of operation seems like it would be useful across CloudStack. Therefore, I would further suggest creating a utility method in utils (e.g. ``copyMapToImmutableMap``) to wrap this operation and properly unit test it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r59085130
--- Diff: engine/storage/snapshot/src/org/apache/cloudstack/storage/snapshot/XenserverSnapshotStrategy.java ---
@@ -279,7 +284,52 @@ public boolean deleteSnapshot(Long snapshotId) {
@Override
public boolean revertSnapshot(SnapshotInfo snapshot) {
- throw new CloudRuntimeException("revert Snapshot is not supported");
+ if (canHandle(snapshot,SnapshotOperation.REVERT) == StrategyPriority.CANT_HANDLE) {
+ throw new UnsupportedOperationException("Reverting not supported. Create a template or volume based on the snapshot instead.");
--- End diff --
``UnsupportedOperationException`` means that a method implementation is not supported on a class. It seems that ``CloudRuntimeException`` would be a more appropriate exception to throw under these circumstances.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by mike-tutkowski <gi...@git.apache.org>.
Github user mike-tutkowski commented on the pull request:
https://github.com/apache/cloudstack/pull/1403#issuecomment-207537423
Hi @swill - No problem for me to squash the commits, but are you sure you want to lose the history of the individual commits?
Also, from what I've seen over the years, it seems more common to prefix instance member variables with "_" than to not do so, but I'm not sure if we have a coding guideline around that or not.
Can you clarify, too, if a LGTM has to come from a committer or can it be any dev on the list?
If any dev on the list, we have Syed and I believe Gabriel (although I didn't see a literal LGTM from him, but rather he provided his comments on the code).
Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by mike-tutkowski <gi...@git.apache.org>.
Github user mike-tutkowski commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r63131882
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -55,65 +68,98 @@
import com.cloud.host.Host;
import com.cloud.host.HostVO;
import com.cloud.host.dao.HostDao;
+import com.cloud.host.dao.HostDetailsDao;
import com.cloud.hypervisor.Hypervisor.HypervisorType;
-import com.cloud.org.Cluster;
-import com.cloud.org.Grouping.AllocationState;
-import com.cloud.resource.ResourceState;
import com.cloud.server.ManagementService;
import com.cloud.storage.DataStoreRole;
import com.cloud.storage.DiskOfferingVO;
import com.cloud.storage.SnapshotVO;
import com.cloud.storage.Storage.ImageFormat;
+import com.cloud.storage.VolumeDetailVO;
import com.cloud.storage.VolumeVO;
import com.cloud.storage.dao.DiskOfferingDao;
import com.cloud.storage.dao.SnapshotDao;
import com.cloud.storage.dao.SnapshotDetailsDao;
import com.cloud.storage.dao.SnapshotDetailsVO;
import com.cloud.storage.dao.VolumeDao;
+import com.cloud.storage.dao.VolumeDetailsDao;
import com.cloud.utils.NumbersUtil;
import com.cloud.utils.exception.CloudRuntimeException;
import com.cloud.vm.VirtualMachineManager;
+import com.google.common.base.Preconditions;
+
@Component
public class StorageSystemDataMotionStrategy implements DataMotionStrategy {
private static final Logger s_logger = Logger.getLogger(StorageSystemDataMotionStrategy.class);
+ private static final Random random = new Random(System.nanoTime());
@Inject private AgentManager _agentMgr;
@Inject private ConfigurationDao _configDao;
+ @Inject private DataStoreManager dataStoreMgr;
@Inject private DiskOfferingDao _diskOfferingDao;
+ @Inject private ClusterDao clusterDao;
@Inject private HostDao _hostDao;
+ @Inject private HostDetailsDao hostDetailsDao;
@Inject private ManagementService _mgr;
@Inject private PrimaryDataStoreDao _storagePoolDao;
@Inject private SnapshotDao _snapshotDao;
@Inject private SnapshotDetailsDao _snapshotDetailsDao;
@Inject private VolumeDao _volumeDao;
@Inject private VolumeDataFactory _volumeDataFactory;
+ @Inject private VolumeDetailsDao volumeDetailsDao;
@Inject private VolumeService _volumeService;
@Override
public StrategyPriority canHandle(DataObject srcData, DataObject destData) {
if (srcData instanceof SnapshotInfo) {
- if (canHandle(srcData.getDataStore()) || canHandle(destData.getDataStore())) {
+ if (canHandle(srcData) || canHandle(destData)) {
return StrategyPriority.HIGHEST;
}
}
+ if (srcData instanceof TemplateInfo && destData instanceof VolumeInfo &&
+ (srcData.getDataStore().getId() == destData.getDataStore().getId()) &&
+ (canHandle(srcData) || canHandle(destData))) {
+ // Both source and dest are on the same storage, so just clone them.
+ return StrategyPriority.HIGHEST;
+ }
+
return StrategyPriority.CANT_HANDLE;
}
- private boolean canHandle(DataStore dataStore) {
+ private boolean canHandle(DataObject dataObject) {
+ Preconditions.checkArgument(dataObject != null, "Passing 'null' to dataObject of canHandle(DataObject) is not supported.");
+
+ DataStore dataStore = dataObject.getDataStore();
+
if (dataStore.getRole() == DataStoreRole.Primary) {
Map<String, String> mapCapabilities = dataStore.getDriver().getCapabilities();
- if (mapCapabilities != null) {
+ if (mapCapabilities == null) {
+ return false;
+ }
+
+ if (dataObject instanceof VolumeInfo || dataObject instanceof SnapshotInfo) {
String value = mapCapabilities.get(DataStoreCapabilities.STORAGE_SYSTEM_SNAPSHOT.toString());
- Boolean supportsStorageSystemSnapshots = new Boolean(value);
+ Boolean supportsStorageSystemSnapshots = Boolean.valueOf(value);
if (supportsStorageSystemSnapshots) {
s_logger.info("Using 'StorageSystemDataMotionStrategy'");
return true;
}
+ } else if (dataObject instanceof TemplateInfo) {
+ // If the storage system can clone volumes, we can cache templates on it.
+ String value = mapCapabilities.get(DataStoreCapabilities.CAN_CREATE_VOLUME_FROM_VOLUME.toString());
+ Boolean canCloneVolume = Boolean.valueOf(value);
+
+ if (canCloneVolume) {
+ s_logger.info("Using 'StorageSystemDataMotionStrategy'");
--- End diff --
Done
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by mike-tutkowski <gi...@git.apache.org>.
Github user mike-tutkowski commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r63133626
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -361,59 +600,205 @@ private String getProperty(long snapshotId, String property) {
}
private Map<String, String> getVolumeDetails(VolumeInfo volumeInfo) {
- Map<String, String> sourceDetails = new HashMap<String, String>();
+ Map<String, String> volumeDetails = new HashMap<String, String>();
VolumeVO volumeVO = _volumeDao.findById(volumeInfo.getId());
long storagePoolId = volumeVO.getPoolId();
StoragePoolVO storagePoolVO = _storagePoolDao.findById(storagePoolId);
- sourceDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
- sourceDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
- sourceDetails.put(DiskTO.IQN, volumeVO.get_iScsiName());
+ volumeDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
+ volumeDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
+ volumeDetails.put(DiskTO.IQN, volumeVO.get_iScsiName());
ChapInfo chapInfo = _volumeService.getChapInfo(volumeInfo, volumeInfo.getDataStore());
if (chapInfo != null) {
- sourceDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, chapInfo.getInitiatorUsername());
- sourceDetails.put(DiskTO.CHAP_INITIATOR_SECRET, chapInfo.getInitiatorSecret());
- sourceDetails.put(DiskTO.CHAP_TARGET_USERNAME, chapInfo.getTargetUsername());
- sourceDetails.put(DiskTO.CHAP_TARGET_SECRET, chapInfo.getTargetSecret());
+ volumeDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, chapInfo.getInitiatorUsername());
+ volumeDetails.put(DiskTO.CHAP_INITIATOR_SECRET, chapInfo.getInitiatorSecret());
+ volumeDetails.put(DiskTO.CHAP_TARGET_USERNAME, chapInfo.getTargetUsername());
+ volumeDetails.put(DiskTO.CHAP_TARGET_SECRET, chapInfo.getTargetSecret());
}
- return sourceDetails;
+ return volumeDetails;
}
- public HostVO getHost(long dataStoreId) {
- StoragePoolVO storagePoolVO = _storagePoolDao.findById(dataStoreId);
+ private Map<String, String> getSnapshotDetails(SnapshotInfo snapshotInfo) {
+ Map<String, String> snapshotDetails = new HashMap<String, String>();
+
+ long storagePoolId = snapshotInfo.getDataStore().getId();
+ StoragePoolVO storagePoolVO = _storagePoolDao.findById(storagePoolId);
+
+ snapshotDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
+ snapshotDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
+
+ long snapshotId = snapshotInfo.getId();
+
+ snapshotDetails.put(DiskTO.IQN, getProperty(snapshotId, DiskTO.IQN));
- List<? extends Cluster> clusters = _mgr.searchForClusters(storagePoolVO.getDataCenterId(), new Long(0), Long.MAX_VALUE, HypervisorType.XenServer.toString());
+ snapshotDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, getProperty(snapshotId, DiskTO.CHAP_INITIATOR_USERNAME));
+ snapshotDetails.put(DiskTO.CHAP_INITIATOR_SECRET, getProperty(snapshotId, DiskTO.CHAP_INITIATOR_SECRET));
+ snapshotDetails.put(DiskTO.CHAP_TARGET_USERNAME, getProperty(snapshotId, DiskTO.CHAP_TARGET_USERNAME));
+ snapshotDetails.put(DiskTO.CHAP_TARGET_SECRET, getProperty(snapshotId, DiskTO.CHAP_TARGET_SECRET));
- if (clusters == null) {
- throw new CloudRuntimeException("Unable to locate an applicable cluster");
+ return snapshotDetails;
+ }
+
+ private HostVO getHost(SnapshotInfo snapshotInfo) {
+ HostVO hostVO = getHost(snapshotInfo.getDataCenterId(), true);
+
+ if (hostVO == null) {
+ hostVO = getHost(snapshotInfo.getDataCenterId(), false);
+
+ if (hostVO == null) {
+ throw new CloudRuntimeException("Unable to locate an applicable host in data center with ID = " + snapshotInfo.getDataCenterId());
+ }
}
- for (Cluster cluster : clusters) {
- if (cluster.getAllocationState() == AllocationState.Enabled) {
- List<HostVO> hosts = _hostDao.findByClusterId(cluster.getId());
+ return hostVO;
+ }
+
+ private HostVO getHost(Long zoneId, boolean computeClusterMustSupportResign) {
+ Preconditions.checkArgument(zoneId != null, "Zone ID cannot be null.");
- if (hosts != null) {
- for (HostVO host : hosts) {
- if (host.getResourceState() == ResourceState.Enabled) {
- return host;
- }
- }
+ List<HostVO> hosts = _hostDao.listByDataCenterIdAndHypervisorType(zoneId, HypervisorType.XenServer);
+
+ if (hosts == null) {
+ return null;
+ }
+
+ List<Long> clustersToSkip = new ArrayList<>();
+
+ Collections.shuffle(hosts, random);
+
+ for (HostVO host : hosts) {
+ if (computeClusterMustSupportResign) {
+ long clusterId = host.getClusterId();
+
+ if (clustersToSkip.contains(clusterId)) {
+ continue;
+ }
+
+ if (clusterDao.computeWhetherClusterSupportsResigning(clusterId)) {
+ return host;
+ }
+ else {
+ clustersToSkip.add(clusterId);
}
}
+ else {
+ return host;
+ }
}
- throw new CloudRuntimeException("Unable to locate an applicable cluster");
+ return null;
}
@Override
public void copyAsync(Map<VolumeInfo, DataStore> volumeMap, VirtualMachineTO vmTo, Host srcHost, Host destHost, AsyncCompletionCallback<CopyCommandResult> callback) {
CopyCommandResult result = new CopyCommandResult(null, null);
+
result.setResult("Unsupported operation requested for copying data.");
+
callback.complete(result);
}
+
+ private Map<String, String> getDetails(DataObject dataObj) {
+ if (dataObj instanceof VolumeInfo) {
+ return getVolumeDetails((VolumeInfo)dataObj);
+ }
+ else if (dataObj instanceof SnapshotInfo) {
+ return getSnapshotDetails((SnapshotInfo)dataObj);
+ }
+
+ throw new CloudRuntimeException("'dataObj' must be of type 'VolumeInfo' or 'SnapshotInfo'.");
+ }
+
+ private CopyCmdAnswer performResignature(DataObject dataObj, HostVO hostVO) {
+ return performResignature(dataObj, hostVO, false);
+ }
+
+ private CopyCmdAnswer performResignature(DataObject dataObj, HostVO hostVO, boolean keepGrantedAccess) {
+ long storagePoolId = dataObj.getDataStore().getId();
+ DataStore dataStore = dataStoreMgr.getDataStore(storagePoolId, DataStoreRole.Primary);
+
+ Map<String, String> details = getDetails(dataObj);
+
+ ResignatureCommand command = new ResignatureCommand(details);
+
+ ResignatureAnswer answer = null;
+
+ try {
+ _volumeService.grantAccess(dataObj, hostVO, dataStore);
+
+ answer = (ResignatureAnswer)_agentMgr.send(hostVO.getId(), command);
+ }
+ catch (AgentUnavailableException | OperationTimedoutException ex) {
+ keepGrantedAccess = false;
+
+ String msg = "Failed to revoke access to the volume from the hosts : ";
--- End diff --
Done
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r63111964
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -255,99 +448,153 @@ private Void handleCreateVolumeFromSnapshotBothOnStorageSystem(SnapshotInfo snap
VolumeApiResult result = future.get();
+ if (volumeDetail != null) {
+ _volumeDetailsDao.remove(volumeDetail.getId());
+ }
+
if (result.isFailed()) {
s_logger.debug("Failed to create a volume: " + result.getResult());
throw new CloudRuntimeException(result.getResult());
}
- }
- catch (Exception ex) {
- throw new CloudRuntimeException(ex.getMessage());
- }
- volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
+ volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
- volumeInfo.processEvent(Event.MigrationRequested);
+ volumeInfo.processEvent(Event.MigrationRequested);
- volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
+ volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
- HostVO hostVO = getHost(snapshotInfo.getDataStore().getId());
+ if (useCloning) {
+ copyCmdAnswer = performResignature(volumeInfo, hostVO);
+ }
+ else {
+ // asking for a XenServer host here so we don't always prefer to use XenServer hosts that support resigning
+ // even when we don't need those hosts to do this kind of copy work
+ hostVO = getHost(snapshotInfo.getDataCenterId(), false);
- String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
- int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
- CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), volumeInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+ copyCmdAnswer = performCopyOfVdi(volumeInfo, snapshotInfo, hostVO);
+ }
- CopyCmdAnswer copyCmdAnswer = null;
+ if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
+ if (copyCmdAnswer != null && !StringUtils.isEmpty(copyCmdAnswer.getDetails())) {
+ errMsg = copyCmdAnswer.getDetails();
+ }
+ else {
+ errMsg = "Unable to perform host-side operation";
+ }
+ }
+ }
+ catch (Exception ex) {
+ errMsg = ex.getMessage() != null ? ex.getMessage() : "Copy operation failed";
--- End diff --
@mike-tutkowski would it be possible to add more context information to this error message?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by mike-tutkowski <gi...@git.apache.org>.
Github user mike-tutkowski commented on the pull request:
https://github.com/apache/cloudstack/pull/1403#issuecomment-219146275
@jburwell @DaanHoogland I have a solution for quickly looking up if a cluster supports resigning that I think we'll all be happy with.
Upon a host connecting to the management server is when I check to see if the host supports resigning (and update the host_details table).
I added logic to this connection code to not only update the host_details table with info about resigning, but also the cluster_details table about resigning.
A host connecting to the management server should not be a frequent occurrence, so I believe it's OK at this point to run through the list of hosts in the cluster of the connecting host and see if all of those hosts support resigning. If they do, then I update the cluster_details table that the cluster in question supports resigning.
I also changed the logic to not bother to add a row to either the host_details table or the cluster_details table if the "supportsResign" property would be false (only a true value is stored now). We can then save space by understanding that a missing "supportsResign" property for a host or cluster indicates false (I was proceeding under that assumption anyways).
When the time comes to ask the cluster if it supports resigning, it's a simple matter of looking for the "supportsResign" property in the cluster_details table.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by DaanHoogland <gi...@git.apache.org>.
Github user DaanHoogland commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r63083577
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -255,99 +473,123 @@ private Void handleCreateVolumeFromSnapshotBothOnStorageSystem(SnapshotInfo snap
VolumeApiResult result = future.get();
+ if (volumeDetail != null) {
+ _volumeDetailsDao.remove(volumeDetail.getId());
+ }
+
if (result.isFailed()) {
- s_logger.debug("Failed to create a volume: " + result.getResult());
+ s_logger.warn("Failed to create a volume: " + result.getResult());
throw new CloudRuntimeException(result.getResult());
}
- }
- catch (Exception ex) {
- throw new CloudRuntimeException(ex.getMessage());
- }
- volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
+ volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
- volumeInfo.processEvent(Event.MigrationRequested);
+ volumeInfo.processEvent(Event.MigrationRequested);
- volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
+ volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
- HostVO hostVO = getHost(snapshotInfo.getDataStore().getId());
-
- String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
- int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
- CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), volumeInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+ if (useCloning) {
+ copyCmdAnswer = performResignature(volumeInfo, hostVO);
+ }
+ else {
+ // asking for a XenServer host here so we don't always prefer to use XenServer hosts that support resigning
+ // even when we don't need those hosts to do this kind of copy work
+ hostVO = getHost(snapshotInfo.getDataCenterId(), false);
- CopyCmdAnswer copyCmdAnswer = null;
+ copyCmdAnswer = performCopyOfVdi(volumeInfo, snapshotInfo, hostVO);
+ }
- try {
- _volumeService.grantAccess(snapshotInfo, hostVO, snapshotInfo.getDataStore());
- _volumeService.grantAccess(volumeInfo, hostVO, volumeInfo.getDataStore());
+ if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
+ if (copyCmdAnswer != null && !StringUtils.isEmpty(copyCmdAnswer.getDetails())) {
+ errMsg = copyCmdAnswer.getDetails();
+ }
+ else {
+ errMsg = "Unable to create volume from snapshot";
+ }
+ }
+ }
+ catch (Exception ex) {
+ errMsg = ex.getMessage() != null ? ex.getMessage() : "Copy operation failed";
+ }
- Map<String, String> srcDetails = getSnapshotDetails(_storagePoolDao.findById(snapshotInfo.getDataStore().getId()), snapshotInfo);
+ CopyCommandResult result = new CopyCommandResult(null, copyCmdAnswer);
- copyCommand.setOptions(srcDetails);
+ result.setResult(errMsg);
- Map<String, String> destDetails = getVolumeDetails(volumeInfo);
+ callback.complete(result);
+ }
- copyCommand.setOptions2(destDetails);
+ /**
+ * If the underlying storage system is making use of read-only snapshots, this gives the storage system the opportunity to
+ * create a volume from the snapshot so that we can copy the VHD file that should be inside of the snapshot to secondary storage.
+ *
+ * The resultant volume must be writable because we need to resign the SR and the VDI that should be inside of it before we copy
+ * the VHD file to secondary storage.
+ *
+ * If the storage system is using writable snapshots, then nothing need be done by that storage system here because we can just
+ * resign the SR and the VDI that should be inside of the snapshot before copying the VHD file to secondary storage.
+ */
+ private void createVolumeFromSnapshot(HostVO hostVO, SnapshotInfo snapshotInfo, boolean keepGrantedAccess) {
+ SnapshotDetailsVO snapshotDetails = handleSnapshotDetails(snapshotInfo.getId(), "tempVolume", "create");
- copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
- }
- catch (Exception ex) {
- throw new CloudRuntimeException(ex.getMessage());
+ try {
+ snapshotInfo.getDataStore().getDriver().createAsync(snapshotInfo.getDataStore(), snapshotInfo, null);
}
finally {
- try {
- _volumeService.revokeAccess(snapshotInfo, hostVO, snapshotInfo.getDataStore());
- }
- catch (Exception ex) {
- s_logger.debug(ex.getMessage(), ex);
- }
-
- try {
- _volumeService.revokeAccess(volumeInfo, hostVO, volumeInfo.getDataStore());
- }
- catch (Exception ex) {
- s_logger.debug(ex.getMessage(), ex);
- }
+ _snapshotDetailsDao.remove(snapshotDetails.getId());
}
- String errMsg = null;
+ CopyCmdAnswer copyCmdAnswer = performResignature(snapshotInfo, hostVO, keepGrantedAccess);
if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
- if (copyCmdAnswer != null && copyCmdAnswer.getDetails() != null && !copyCmdAnswer.getDetails().isEmpty()) {
- errMsg = copyCmdAnswer.getDetails();
+ if (copyCmdAnswer != null && !StringUtils.isEmpty(copyCmdAnswer.getDetails())) {
+ throw new CloudRuntimeException(copyCmdAnswer.getDetails());
}
else {
- errMsg = "Unable to perform host-side operation";
+ throw new CloudRuntimeException("Unable to create volume from snapshot");
}
}
+ }
- CopyCommandResult result = new CopyCommandResult(null, copyCmdAnswer);
+ /**
+ * If the underlying storage system needed to create a volume from a snapshot for createVolumeFromSnapshot(HostVO, SnapshotInfo), then
+ * this is its opportunity to delete that temporary volume and restore properties in snapshot_details to the way they were before the
--- End diff --
Well, in the comment you speak of opportunity. It seems this is not the opportunity but the grabbing of an opportunity that is found at some other place in the code. In other words, I don't understand the use of the word hence I don't understand the comment.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r63112811
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -361,59 +600,205 @@ private String getProperty(long snapshotId, String property) {
}
private Map<String, String> getVolumeDetails(VolumeInfo volumeInfo) {
- Map<String, String> sourceDetails = new HashMap<String, String>();
+ Map<String, String> volumeDetails = new HashMap<String, String>();
VolumeVO volumeVO = _volumeDao.findById(volumeInfo.getId());
long storagePoolId = volumeVO.getPoolId();
StoragePoolVO storagePoolVO = _storagePoolDao.findById(storagePoolId);
- sourceDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
- sourceDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
- sourceDetails.put(DiskTO.IQN, volumeVO.get_iScsiName());
+ volumeDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
+ volumeDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
+ volumeDetails.put(DiskTO.IQN, volumeVO.get_iScsiName());
ChapInfo chapInfo = _volumeService.getChapInfo(volumeInfo, volumeInfo.getDataStore());
if (chapInfo != null) {
- sourceDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, chapInfo.getInitiatorUsername());
- sourceDetails.put(DiskTO.CHAP_INITIATOR_SECRET, chapInfo.getInitiatorSecret());
- sourceDetails.put(DiskTO.CHAP_TARGET_USERNAME, chapInfo.getTargetUsername());
- sourceDetails.put(DiskTO.CHAP_TARGET_SECRET, chapInfo.getTargetSecret());
+ volumeDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, chapInfo.getInitiatorUsername());
+ volumeDetails.put(DiskTO.CHAP_INITIATOR_SECRET, chapInfo.getInitiatorSecret());
+ volumeDetails.put(DiskTO.CHAP_TARGET_USERNAME, chapInfo.getTargetUsername());
+ volumeDetails.put(DiskTO.CHAP_TARGET_SECRET, chapInfo.getTargetSecret());
}
- return sourceDetails;
+ return volumeDetails;
}
- public HostVO getHost(long dataStoreId) {
- StoragePoolVO storagePoolVO = _storagePoolDao.findById(dataStoreId);
+ private Map<String, String> getSnapshotDetails(SnapshotInfo snapshotInfo) {
+ Map<String, String> snapshotDetails = new HashMap<String, String>();
+
+ long storagePoolId = snapshotInfo.getDataStore().getId();
+ StoragePoolVO storagePoolVO = _storagePoolDao.findById(storagePoolId);
+
+ snapshotDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
+ snapshotDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
+
+ long snapshotId = snapshotInfo.getId();
+
+ snapshotDetails.put(DiskTO.IQN, getProperty(snapshotId, DiskTO.IQN));
- List<? extends Cluster> clusters = _mgr.searchForClusters(storagePoolVO.getDataCenterId(), new Long(0), Long.MAX_VALUE, HypervisorType.XenServer.toString());
+ snapshotDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, getProperty(snapshotId, DiskTO.CHAP_INITIATOR_USERNAME));
+ snapshotDetails.put(DiskTO.CHAP_INITIATOR_SECRET, getProperty(snapshotId, DiskTO.CHAP_INITIATOR_SECRET));
+ snapshotDetails.put(DiskTO.CHAP_TARGET_USERNAME, getProperty(snapshotId, DiskTO.CHAP_TARGET_USERNAME));
+ snapshotDetails.put(DiskTO.CHAP_TARGET_SECRET, getProperty(snapshotId, DiskTO.CHAP_TARGET_SECRET));
- if (clusters == null) {
- throw new CloudRuntimeException("Unable to locate an applicable cluster");
+ return snapshotDetails;
+ }
+
+ private HostVO getHost(SnapshotInfo snapshotInfo) {
+ HostVO hostVO = getHost(snapshotInfo.getDataCenterId(), true);
+
+ if (hostVO == null) {
+ hostVO = getHost(snapshotInfo.getDataCenterId(), false);
+
+ if (hostVO == null) {
+ throw new CloudRuntimeException("Unable to locate an applicable host in data center with ID = " + snapshotInfo.getDataCenterId());
+ }
}
- for (Cluster cluster : clusters) {
- if (cluster.getAllocationState() == AllocationState.Enabled) {
- List<HostVO> hosts = _hostDao.findByClusterId(cluster.getId());
+ return hostVO;
+ }
+
+ private HostVO getHost(Long zoneId, boolean computeClusterMustSupportResign) {
+ Preconditions.checkArgument(zoneId != null, "Zone ID cannot be null.");
- if (hosts != null) {
- for (HostVO host : hosts) {
- if (host.getResourceState() == ResourceState.Enabled) {
- return host;
- }
- }
+ List<HostVO> hosts = _hostDao.listByDataCenterIdAndHypervisorType(zoneId, HypervisorType.XenServer);
+
+ if (hosts == null) {
+ return null;
+ }
+
+ List<Long> clustersToSkip = new ArrayList<>();
+
+ Collections.shuffle(hosts, random);
+
+ for (HostVO host : hosts) {
+ if (computeClusterMustSupportResign) {
+ long clusterId = host.getClusterId();
+
+ if (clustersToSkip.contains(clusterId)) {
+ continue;
+ }
+
+ if (clusterDao.computeWhetherClusterSupportsResigning(clusterId)) {
+ return host;
+ }
+ else {
+ clustersToSkip.add(clusterId);
}
}
+ else {
+ return host;
+ }
}
- throw new CloudRuntimeException("Unable to locate an applicable cluster");
+ return null;
}
@Override
public void copyAsync(Map<VolumeInfo, DataStore> volumeMap, VirtualMachineTO vmTo, Host srcHost, Host destHost, AsyncCompletionCallback<CopyCommandResult> callback) {
CopyCommandResult result = new CopyCommandResult(null, null);
+
result.setResult("Unsupported operation requested for copying data.");
+
callback.complete(result);
}
+
+ private Map<String, String> getDetails(DataObject dataObj) {
+ if (dataObj instanceof VolumeInfo) {
+ return getVolumeDetails((VolumeInfo)dataObj);
+ }
+ else if (dataObj instanceof SnapshotInfo) {
+ return getSnapshotDetails((SnapshotInfo)dataObj);
+ }
+
+ throw new CloudRuntimeException("'dataObj' must be of type 'VolumeInfo' or 'SnapshotInfo'.");
+ }
+
+ private CopyCmdAnswer performResignature(DataObject dataObj, HostVO hostVO) {
+ return performResignature(dataObj, hostVO, false);
+ }
+
+ private CopyCmdAnswer performResignature(DataObject dataObj, HostVO hostVO, boolean keepGrantedAccess) {
+ long storagePoolId = dataObj.getDataStore().getId();
+ DataStore dataStore = dataStoreMgr.getDataStore(storagePoolId, DataStoreRole.Primary);
+
+ Map<String, String> details = getDetails(dataObj);
+
+ ResignatureCommand command = new ResignatureCommand(details);
+
+ ResignatureAnswer answer = null;
+
+ try {
+ _volumeService.grantAccess(dataObj, hostVO, dataStore);
+
+ answer = (ResignatureAnswer)_agentMgr.send(hostVO.getId(), command);
+ }
+ catch (AgentUnavailableException | OperationTimedoutException ex) {
+ keepGrantedAccess = false;
+
+ String msg = "Failed to revoke access to the volume from the hosts : ";
--- End diff --
Please include the volume and host ids in the error message to assist operational debugging.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by DaanHoogland <gi...@git.apache.org>.
Github user DaanHoogland commented on the pull request:
https://github.com/apache/cloudstack/pull/1403#issuecomment-218865225
LGTM with one extra comment: This is adding a much wanted feature and not worsening the code base we have but some bad patterns are being maintained that we should divert from. Hope you agree @mike-tutkowski
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by DaanHoogland <gi...@git.apache.org>.
Github user DaanHoogland commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r62885313
--- Diff: plugins/hypervisors/xenserver/src/com/cloud/hypervisor/xenserver/resource/CitrixResourceBase.java ---
@@ -164,9 +165,12 @@
*
*/
public abstract class CitrixResourceBase implements ServerResource, HypervisorResource, VirtualRouterDeployer {
-
+ /**
+ * RELVMOISCSI = used for resigning metadata (like SR UUID and VDI UUID when a
+ * particular storage manager is installed on a XenServer host (for back-end snapshots to work))
+ */
--- End diff --
I would put the value javadoc on the value and describe only the whole enum here. (PR coming to that end on another enum, shortly)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by DaanHoogland <gi...@git.apache.org>.
Github user DaanHoogland commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r62880347
--- Diff: engine/storage/snapshot/src/org/apache/cloudstack/storage/snapshot/StorageSystemSnapshotStrategy.java ---
@@ -347,38 +395,87 @@ private String getProperty(long snapshotId, String property) {
return null;
}
- private HostVO getHost(Long hostId, VolumeVO volumeVO) {
- HostVO hostVO = _hostDao.findById(hostId);
+ private HostVO getHost(long volumeId) {
+ VolumeVO volumeVO = _volumeDao.findById(volumeId);
+
+ Long vmInstanceId = volumeVO.getInstanceId();
+ VMInstanceVO vmInstanceVO = _vmInstanceDao.findById(vmInstanceId);
+
+ Long hostId = null;
+
+ // if the volume to snapshot is associated with a VM
+ if (vmInstanceVO != null) {
+ hostId = vmInstanceVO.getHostId();
+
+ // if the VM is not associated with a host
+ if (hostId == null) {
+ hostId = vmInstanceVO.getLastHostId();
+ }
+ }
+
+ return getHost(volumeVO.getDataCenterId(), hostId);
+ }
+
+ private HostVO getHost(long zoneId, Long hostId) {
+ HostVO hostVO = getHost(zoneId, true);
if (hostVO != null) {
return hostVO;
}
- // pick a host in any XenServer cluster that's in the applicable zone
+ hostVO = _hostDao.findById(hostId);
- long zoneId = volumeVO.getDataCenterId();
+ if (hostVO != null) {
+ return hostVO;
+ }
+
+ hostVO = getHost(zoneId, false);
- List<? extends Cluster> clusters = _mgr.searchForClusters(zoneId, new Long(0), Long.MAX_VALUE, HypervisorType.XenServer.toString());
+ if (hostVO != null) {
+ return hostVO;
+ }
+
+ throw new CloudRuntimeException("Unable to locate an applicable host");
+ }
+
+ private HostVO getHost(long zoneId, boolean computeClusterMustSupportResign) {
+ List<? extends Cluster> clusters = _mgr.searchForClusters(zoneId, 0L, Long.MAX_VALUE, HypervisorType.XenServer.toString());
if (clusters == null) {
- throw new CloudRuntimeException("Unable to locate an applicable cluster");
+ clusters = new ArrayList<>();
}
+ Collections.shuffle(clusters, new Random(System.nanoTime()));
+
+ clusters:
for (Cluster cluster : clusters) {
if (cluster.getAllocationState() == AllocationState.Enabled) {
List<HostVO> hosts = _hostDao.findByClusterId(cluster.getId());
if (hosts != null) {
+ Collections.shuffle(hosts, new Random(System.nanoTime()));
+
for (HostVO host : hosts) {
if (host.getResourceState() == ResourceState.Enabled) {
- return host;
+ if (computeClusterMustSupportResign) {
+ if (_clusterDao.computeClusterSupportsResign(cluster.getId())) {
+ return host;
+ }
+ else {
+ // no other host in the cluster in question should be able to satisfy our requirements here, so move on to the next cluster
+ continue clusters;
+ }
+ }
+ else {
+ return host;
+ }
}
}
}
}
}
- throw new CloudRuntimeException("Unable to locate an applicable cluster");
+ return null;
--- End diff --
me no like, maybe a local private checked exception to have it handled in the top getHost method?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r59081095
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -361,59 +608,225 @@ private String getProperty(long snapshotId, String property) {
}
private Map<String, String> getVolumeDetails(VolumeInfo volumeInfo) {
- Map<String, String> sourceDetails = new HashMap<String, String>();
+ Map<String, String> volumeDetails = new HashMap<String, String>();
VolumeVO volumeVO = _volumeDao.findById(volumeInfo.getId());
long storagePoolId = volumeVO.getPoolId();
StoragePoolVO storagePoolVO = _storagePoolDao.findById(storagePoolId);
- sourceDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
- sourceDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
- sourceDetails.put(DiskTO.IQN, volumeVO.get_iScsiName());
+ volumeDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
+ volumeDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
+ volumeDetails.put(DiskTO.IQN, volumeVO.get_iScsiName());
ChapInfo chapInfo = _volumeService.getChapInfo(volumeInfo, volumeInfo.getDataStore());
if (chapInfo != null) {
- sourceDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, chapInfo.getInitiatorUsername());
- sourceDetails.put(DiskTO.CHAP_INITIATOR_SECRET, chapInfo.getInitiatorSecret());
- sourceDetails.put(DiskTO.CHAP_TARGET_USERNAME, chapInfo.getTargetUsername());
- sourceDetails.put(DiskTO.CHAP_TARGET_SECRET, chapInfo.getTargetSecret());
+ volumeDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, chapInfo.getInitiatorUsername());
+ volumeDetails.put(DiskTO.CHAP_INITIATOR_SECRET, chapInfo.getInitiatorSecret());
+ volumeDetails.put(DiskTO.CHAP_TARGET_USERNAME, chapInfo.getTargetUsername());
+ volumeDetails.put(DiskTO.CHAP_TARGET_SECRET, chapInfo.getTargetSecret());
+ }
+
+ return volumeDetails;
+ }
+
+ private Map<String, String> getSnapshotDetails(SnapshotInfo snapshotInfo) {
+ Map<String, String> snapshotDetails = new HashMap<String, String>();
+
+ long storagePoolId = snapshotInfo.getDataStore().getId();
+ StoragePoolVO storagePoolVO = _storagePoolDao.findById(storagePoolId);
+
+ snapshotDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
+ snapshotDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
+
+ long snapshotId = snapshotInfo.getId();
+
+ snapshotDetails.put(DiskTO.IQN, getProperty(snapshotId, DiskTO.IQN));
+
+ snapshotDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, getProperty(snapshotId, DiskTO.CHAP_INITIATOR_USERNAME));
+ snapshotDetails.put(DiskTO.CHAP_INITIATOR_SECRET, getProperty(snapshotId, DiskTO.CHAP_INITIATOR_SECRET));
+ snapshotDetails.put(DiskTO.CHAP_TARGET_USERNAME, getProperty(snapshotId, DiskTO.CHAP_TARGET_USERNAME));
+ snapshotDetails.put(DiskTO.CHAP_TARGET_SECRET, getProperty(snapshotId, DiskTO.CHAP_TARGET_SECRET));
+
+ return snapshotDetails;
+ }
+
+ private HostVO getHost(SnapshotInfo snapshotInfo) {
+ HostVO hostVO = getHost(snapshotInfo.getDataCenterId(), true);
+
+ if (hostVO == null) {
+ hostVO = getHost(snapshotInfo.getDataCenterId(), false);
+
+ if (hostVO == null) {
+ throw new CloudRuntimeException("Unable to locate an applicable host");
+ }
}
- return sourceDetails;
+ return hostVO;
}
- public HostVO getHost(long dataStoreId) {
- StoragePoolVO storagePoolVO = _storagePoolDao.findById(dataStoreId);
+ private HostVO getHost(Long zoneId, boolean computeClusterMustSupportResign) {
+ if (zoneId == null) {
+ throw new CloudRuntimeException("Zone ID cannot be null.");
+ }
- List<? extends Cluster> clusters = _mgr.searchForClusters(storagePoolVO.getDataCenterId(), new Long(0), Long.MAX_VALUE, HypervisorType.XenServer.toString());
+ List<? extends Cluster> clusters = _mgr.searchForClusters(zoneId, new Long(0), Long.MAX_VALUE, HypervisorType.XenServer.toString());
if (clusters == null) {
- throw new CloudRuntimeException("Unable to locate an applicable cluster");
+ clusters = new ArrayList<>();
}
+ Collections.shuffle(clusters, new Random(System.nanoTime()));
+
+ clusters:
for (Cluster cluster : clusters) {
if (cluster.getAllocationState() == AllocationState.Enabled) {
List<HostVO> hosts = _hostDao.findByClusterId(cluster.getId());
if (hosts != null) {
+ Collections.shuffle(hosts, new Random(System.nanoTime()));
+
for (HostVO host : hosts) {
if (host.getResourceState() == ResourceState.Enabled) {
- return host;
+ if (computeClusterMustSupportResign) {
+ if (computeClusterSupportsResign(cluster.getId())) {
+ return host;
+ }
+ else {
+ // no other host in the cluster in question should be able to satisfy our requirements here, so move on to the next cluster
+ continue clusters;
+ }
+ }
+ else {
+ return host;
+ }
}
}
}
}
}
- throw new CloudRuntimeException("Unable to locate an applicable cluster");
+ return null;
}
@Override
public void copyAsync(Map<VolumeInfo, DataStore> volumeMap, VirtualMachineTO vmTo, Host srcHost, Host destHost, AsyncCompletionCallback<CopyCommandResult> callback) {
CopyCommandResult result = new CopyCommandResult(null, null);
+
result.setResult("Unsupported operation requested for copying data.");
+
callback.complete(result);
}
+
+ private Map<String, String> getDetails(DataObject dataObj) {
+ if (dataObj instanceof VolumeInfo) {
+ return getVolumeDetails((VolumeInfo)dataObj);
+ }
+ else if (dataObj instanceof SnapshotInfo) {
+ return getSnapshotDetails((SnapshotInfo)dataObj);
+ }
--- End diff --
Should we consider a new category interface such ``Extensible`` that defines the ``getDetails`` method? We have this pattern all over the codebase, and it seems like it would greatly simplify able to work with objects in those terms. Done properly, we could phase it in.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by DaanHoogland <gi...@git.apache.org>.
Github user DaanHoogland commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r62881271
--- Diff: engine/storage/volume/src/org/apache/cloudstack/storage/volume/VolumeServiceImpl.java ---
@@ -554,6 +574,51 @@ protected Void managedCopyBaseImageCallback(AsyncCallbackDispatcher<VolumeServic
return null;
}
+ protected Void createManagedTemplateImageCallback(AsyncCallbackDispatcher<VolumeServiceImpl, CreateCmdResult> callback, CreateVolumeContext<CreateCmdResult> context) {
+ CreateCmdResult result = callback.getResult();
+ VolumeApiResult res = new VolumeApiResult(null);
+
+ res.setResult(result.getResult());
+
+ AsyncCallFuture<VolumeApiResult> future = context.getFuture();
+ DataObject templateOnPrimaryStoreObj = context.getVolume();
+
+ if (result.isSuccess()) {
+ ((TemplateObject)templateOnPrimaryStoreObj).setInstallPath(result.getPath());
+ templateOnPrimaryStoreObj.processEvent(Event.OperationSuccessed, result.getAnswer());
+ }
+ else {
+ templateOnPrimaryStoreObj.processEvent(Event.OperationFailed);
+ }
+
+ future.complete(res);
+
+ return null;
+ }
+
+ protected Void copyManagedTemplateCallback(AsyncCallbackDispatcher<VolumeServiceImpl, CopyCommandResult> callback, CreateBaseImageContext<VolumeApiResult> context) {
--- End diff --
same here, why Void and returning a null? returning null is a bad smell to me.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by DaanHoogland <gi...@git.apache.org>.
Github user DaanHoogland commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r62886608
--- Diff: plugins/hypervisors/xenserver/src/com/cloud/hypervisor/xenserver/resource/CitrixResourceBase.java ---
@@ -2351,17 +2376,52 @@ public SR getIscsiSR(final Connection conn, final String srNameLabel, final Stri
throw new CloudRuntimeException(msg, e);
}
}
+
deviceConfig.put("SCSIid", scsiid);
- final String result = SR.probe(conn, host, deviceConfig, type, smConfig);
+ String result = SR.probe(conn, host, deviceConfig, type, smConfig);
+
String pooluuid = null;
+
if (result.indexOf("<UUID>") != -1) {
pooluuid = result.substring(result.indexOf("<UUID>") + 6, result.indexOf("</UUID>")).trim();
}
if (pooluuid == null || pooluuid.length() != 36) {
sr = SR.create(conn, host, deviceConfig, new Long(0), srNameLabel, srNameLabel, type, "user", true, smConfig);
- } else {
+ }
+ else {
+ if (resignature) {
--- End diff --
maybe a method createIfResignable()? thsi file is a blob of blobs, let's get to reducing the problem called CitrixResourceBase
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r63111670
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -172,78 +258,207 @@ private void validate(SnapshotInfo snapshotInfo) {
}
}
- private Void handleCreateTemplateFromSnapshot(SnapshotInfo snapshotInfo, TemplateInfo templateInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
+ private boolean usingBackendSnapshotFor(SnapshotInfo snapshotInfo) {
+ String property = getProperty(snapshotInfo.getId(), "takeSnapshot");
+
+ return Boolean.parseBoolean(property);
+ }
+
+ private void handleCreateTemplateFromSnapshot(SnapshotInfo snapshotInfo, TemplateInfo templateInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
try {
snapshotInfo.processEvent(Event.CopyingRequested);
}
catch (Exception ex) {
throw new CloudRuntimeException("This snapshot is not currently in a state where it can be used to create a template.");
}
- HostVO hostVO = getHost(snapshotInfo.getDataStore().getId());
- DataStore srcDataStore = snapshotInfo.getDataStore();
+ HostVO hostVO = getHost(snapshotInfo);
- String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
- int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
- CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), templateInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+ boolean usingBackendSnapshot = usingBackendSnapshotFor(snapshotInfo);
+ boolean computeClusterSupportsResign = clusterDao.computeWhetherClusterSupportsResigning(hostVO.getClusterId());
- String errMsg = null;
-
- CopyCmdAnswer copyCmdAnswer = null;
+ if (usingBackendSnapshot && !computeClusterSupportsResign) {
+ throw new CloudRuntimeException("Unable to locate an applicable host with which to perform a resignature operation");
+ }
try {
- _volumeService.grantAccess(snapshotInfo, hostVO, srcDataStore);
+ if (usingBackendSnapshot) {
+ createVolumeFromSnapshot(hostVO, snapshotInfo, true);
+ }
- Map<String, String> srcDetails = getSnapshotDetails(_storagePoolDao.findById(srcDataStore.getId()), snapshotInfo);
+ DataStore srcDataStore = snapshotInfo.getDataStore();
- copyCommand.setOptions(srcDetails);
+ String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
+ int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
+ CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), templateInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+
+ String errMsg = null;
+
+ CopyCmdAnswer copyCmdAnswer = null;
- copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
- }
- catch (Exception ex) {
- throw new CloudRuntimeException(ex.getMessage());
- }
- finally {
try {
- _volumeService.revokeAccess(snapshotInfo, hostVO, srcDataStore);
+ // If we are using a back-end snapshot, then we should still have access to it from the hosts in the cluster that hostVO is in
+ // (because we passed in true as the third parameter to createVolumeFromSnapshot above).
+ if (usingBackendSnapshot == false) {
+ _volumeService.grantAccess(snapshotInfo, hostVO, srcDataStore);
+ }
+
+ Map<String, String> srcDetails = getSnapshotDetails(snapshotInfo);
+
+ copyCommand.setOptions(srcDetails);
+
+ copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
}
- catch (Exception ex) {
- s_logger.debug(ex.getMessage(), ex);
+ catch (AgentUnavailableException | OperationTimedoutException ex) {
+ throw new CloudRuntimeException("Failed to create template from snapshot : " + ex.getMessage());
--- End diff --
Please add the snapshotId to the error message to assist operational debugging.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r59075048
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -172,78 +232,211 @@ private void validate(SnapshotInfo snapshotInfo) {
}
}
- private Void handleCreateTemplateFromSnapshot(SnapshotInfo snapshotInfo, TemplateInfo templateInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
+ private boolean usingBackendSnapshotFor(SnapshotInfo snapshotInfo) {
+ String property = getProperty(snapshotInfo.getId(), "takeSnapshot");
+
+ return Boolean.parseBoolean(property);
+ }
+
+ private void handleCreateTemplateFromSnapshot(SnapshotInfo snapshotInfo, TemplateInfo templateInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
try {
snapshotInfo.processEvent(Event.CopyingRequested);
}
catch (Exception ex) {
throw new CloudRuntimeException("This snapshot is not currently in a state where it can be used to create a template.");
}
- HostVO hostVO = getHost(snapshotInfo.getDataStore().getId());
- DataStore srcDataStore = snapshotInfo.getDataStore();
-
- String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
- int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
- CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), templateInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+ HostVO hostVO = getHost(snapshotInfo);
- String errMsg = null;
+ boolean usingBackendSnapshot = usingBackendSnapshotFor(snapshotInfo);
+ boolean computeClusterSupportsResign = computeClusterSupportsResign(hostVO.getClusterId());
- CopyCmdAnswer copyCmdAnswer = null;
+ if (usingBackendSnapshot && !computeClusterSupportsResign) {
+ throw new CloudRuntimeException("Unable to locate an applicable host with which to perform a resignature operation");
+ }
try {
- _volumeService.grantAccess(snapshotInfo, hostVO, srcDataStore);
+ if (usingBackendSnapshot) {
+ createVolumeFromSnapshot(hostVO, snapshotInfo, true);
+ }
- Map<String, String> srcDetails = getSnapshotDetails(_storagePoolDao.findById(srcDataStore.getId()), snapshotInfo);
+ DataStore srcDataStore = snapshotInfo.getDataStore();
- copyCommand.setOptions(srcDetails);
+ String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
+ int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
+ CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), templateInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+
+ String errMsg = null;
+
+ CopyCmdAnswer copyCmdAnswer = null;
- copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
- }
- catch (Exception ex) {
- throw new CloudRuntimeException(ex.getMessage());
- }
- finally {
try {
- _volumeService.revokeAccess(snapshotInfo, hostVO, srcDataStore);
+ // If we are using a back-end snapshot, then we should still have access to it from the hosts in the cluster that hostVO is in
+ // (because we passed in true as the third parameter to createVolumeFromSnapshot above).
+ if (usingBackendSnapshot == false) {
+ _volumeService.grantAccess(snapshotInfo, hostVO, srcDataStore);
+ }
+
+ Map<String, String> srcDetails = getSnapshotDetails(snapshotInfo);
+
+ copyCommand.setOptions(srcDetails);
+
+ copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
}
catch (Exception ex) {
- s_logger.debug(ex.getMessage(), ex);
+ throw new CloudRuntimeException(ex.getMessage());
}
+ finally {
+ try {
+ _volumeService.revokeAccess(snapshotInfo, hostVO, srcDataStore);
+ }
+ catch (Exception ex) {
+ s_logger.debug(ex.getMessage(), ex);
+ }
- if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
- if (copyCmdAnswer != null && copyCmdAnswer.getDetails() != null && !copyCmdAnswer.getDetails().isEmpty()) {
- errMsg = copyCmdAnswer.getDetails();
+ if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
+ if (copyCmdAnswer != null && !StringUtils.isEmpty(copyCmdAnswer.getDetails())) {
+ errMsg = copyCmdAnswer.getDetails();
+ }
+ else {
+ errMsg = "Unable to perform host-side operation";
+ }
}
- else {
- errMsg = "Unable to perform host-side operation";
+
+ try {
+ if (errMsg == null) {
+ snapshotInfo.processEvent(Event.OperationSuccessed);
+ }
+ else {
+ snapshotInfo.processEvent(Event.OperationFailed);
+ }
+ }
+ catch (Exception ex) {
+ s_logger.debug(ex.getMessage(), ex);
--- End diff --
Why not log the ignored exception to ``WARN``?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r63112842
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -361,59 +608,225 @@ private String getProperty(long snapshotId, String property) {
}
private Map<String, String> getVolumeDetails(VolumeInfo volumeInfo) {
- Map<String, String> sourceDetails = new HashMap<String, String>();
+ Map<String, String> volumeDetails = new HashMap<String, String>();
VolumeVO volumeVO = _volumeDao.findById(volumeInfo.getId());
long storagePoolId = volumeVO.getPoolId();
StoragePoolVO storagePoolVO = _storagePoolDao.findById(storagePoolId);
- sourceDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
- sourceDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
- sourceDetails.put(DiskTO.IQN, volumeVO.get_iScsiName());
+ volumeDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
+ volumeDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
+ volumeDetails.put(DiskTO.IQN, volumeVO.get_iScsiName());
ChapInfo chapInfo = _volumeService.getChapInfo(volumeInfo, volumeInfo.getDataStore());
if (chapInfo != null) {
- sourceDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, chapInfo.getInitiatorUsername());
- sourceDetails.put(DiskTO.CHAP_INITIATOR_SECRET, chapInfo.getInitiatorSecret());
- sourceDetails.put(DiskTO.CHAP_TARGET_USERNAME, chapInfo.getTargetUsername());
- sourceDetails.put(DiskTO.CHAP_TARGET_SECRET, chapInfo.getTargetSecret());
+ volumeDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, chapInfo.getInitiatorUsername());
+ volumeDetails.put(DiskTO.CHAP_INITIATOR_SECRET, chapInfo.getInitiatorSecret());
+ volumeDetails.put(DiskTO.CHAP_TARGET_USERNAME, chapInfo.getTargetUsername());
+ volumeDetails.put(DiskTO.CHAP_TARGET_SECRET, chapInfo.getTargetSecret());
+ }
+
+ return volumeDetails;
+ }
+
+ private Map<String, String> getSnapshotDetails(SnapshotInfo snapshotInfo) {
+ Map<String, String> snapshotDetails = new HashMap<String, String>();
+
+ long storagePoolId = snapshotInfo.getDataStore().getId();
+ StoragePoolVO storagePoolVO = _storagePoolDao.findById(storagePoolId);
+
+ snapshotDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
+ snapshotDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
+
+ long snapshotId = snapshotInfo.getId();
+
+ snapshotDetails.put(DiskTO.IQN, getProperty(snapshotId, DiskTO.IQN));
+
+ snapshotDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, getProperty(snapshotId, DiskTO.CHAP_INITIATOR_USERNAME));
+ snapshotDetails.put(DiskTO.CHAP_INITIATOR_SECRET, getProperty(snapshotId, DiskTO.CHAP_INITIATOR_SECRET));
+ snapshotDetails.put(DiskTO.CHAP_TARGET_USERNAME, getProperty(snapshotId, DiskTO.CHAP_TARGET_USERNAME));
+ snapshotDetails.put(DiskTO.CHAP_TARGET_SECRET, getProperty(snapshotId, DiskTO.CHAP_TARGET_SECRET));
+
+ return snapshotDetails;
+ }
+
+ private HostVO getHost(SnapshotInfo snapshotInfo) {
+ HostVO hostVO = getHost(snapshotInfo.getDataCenterId(), true);
+
+ if (hostVO == null) {
+ hostVO = getHost(snapshotInfo.getDataCenterId(), false);
+
+ if (hostVO == null) {
+ throw new CloudRuntimeException("Unable to locate an applicable host");
+ }
}
- return sourceDetails;
+ return hostVO;
}
- public HostVO getHost(long dataStoreId) {
- StoragePoolVO storagePoolVO = _storagePoolDao.findById(dataStoreId);
+ private HostVO getHost(Long zoneId, boolean computeClusterMustSupportResign) {
+ if (zoneId == null) {
+ throw new CloudRuntimeException("Zone ID cannot be null.");
+ }
- List<? extends Cluster> clusters = _mgr.searchForClusters(storagePoolVO.getDataCenterId(), new Long(0), Long.MAX_VALUE, HypervisorType.XenServer.toString());
+ List<? extends Cluster> clusters = _mgr.searchForClusters(zoneId, new Long(0), Long.MAX_VALUE, HypervisorType.XenServer.toString());
if (clusters == null) {
- throw new CloudRuntimeException("Unable to locate an applicable cluster");
+ clusters = new ArrayList<>();
}
+ Collections.shuffle(clusters, new Random(System.nanoTime()));
+
+ clusters:
for (Cluster cluster : clusters) {
if (cluster.getAllocationState() == AllocationState.Enabled) {
List<HostVO> hosts = _hostDao.findByClusterId(cluster.getId());
if (hosts != null) {
+ Collections.shuffle(hosts, new Random(System.nanoTime()));
+
for (HostVO host : hosts) {
if (host.getResourceState() == ResourceState.Enabled) {
- return host;
+ if (computeClusterMustSupportResign) {
+ if (computeClusterSupportsResign(cluster.getId())) {
+ return host;
+ }
+ else {
+ // no other host in the cluster in question should be able to satisfy our requirements here, so move on to the next cluster
+ continue clusters;
+ }
+ }
+ else {
+ return host;
+ }
}
}
}
}
}
- throw new CloudRuntimeException("Unable to locate an applicable cluster");
+ return null;
}
@Override
public void copyAsync(Map<VolumeInfo, DataStore> volumeMap, VirtualMachineTO vmTo, Host srcHost, Host destHost, AsyncCompletionCallback<CopyCommandResult> callback) {
CopyCommandResult result = new CopyCommandResult(null, null);
+
result.setResult("Unsupported operation requested for copying data.");
+
callback.complete(result);
}
+
+ private Map<String, String> getDetails(DataObject dataObj) {
+ if (dataObj instanceof VolumeInfo) {
+ return getVolumeDetails((VolumeInfo)dataObj);
+ }
+ else if (dataObj instanceof SnapshotInfo) {
+ return getSnapshotDetails((SnapshotInfo)dataObj);
+ }
+
+ throw new CloudRuntimeException("'dataObj' must be of type 'VolumeInfo' or 'SnapshotInfo'.");
+ }
+
+ private CopyCmdAnswer performResignature(DataObject dataObj, HostVO hostVO) {
+ return performResignature(dataObj, hostVO, false);
+ }
+
+ private CopyCmdAnswer performResignature(DataObject dataObj, HostVO hostVO, boolean keepGrantedAccess) {
+ long storagePoolId = dataObj.getDataStore().getId();
+ DataStore dataStore = _dataStoreMgr.getDataStore(storagePoolId, DataStoreRole.Primary);
+
+ Map<String, String> details = getDetails(dataObj);
+
+ ResignatureCommand command = new ResignatureCommand(details);
+
+ ResignatureAnswer answer = null;
+
+ try {
+ _volumeService.grantAccess(dataObj, hostVO, dataStore);
+
+ answer = (ResignatureAnswer)_agentMgr.send(hostVO.getId(), command);
+ }
+ catch (Exception ex) {
+ keepGrantedAccess = false;
+
+ throw new CloudRuntimeException(ex.getMessage());
+ }
+ finally {
+ if (keepGrantedAccess == false) {
+ try {
+ _volumeService.revokeAccess(dataObj, hostVO, dataStore);
+ }
+ catch (Exception ex) {
+ s_logger.debug(ex.getMessage(), ex);
+ }
+ }
+ }
+
+ if (answer == null || !answer.getResult()) {
+ final String errMsg;
+
+ if (answer != null && answer.getDetails() != null && !answer.getDetails().isEmpty()) {
+ errMsg = answer.getDetails();
+ }
+ else {
+ errMsg = "Unable to perform resignature operation";
--- End diff --
@mike-tutkowski would it be possible to add this contextual information?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by DaanHoogland <gi...@git.apache.org>.
Github user DaanHoogland commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r62874090
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -172,78 +232,211 @@ private void validate(SnapshotInfo snapshotInfo) {
}
}
- private Void handleCreateTemplateFromSnapshot(SnapshotInfo snapshotInfo, TemplateInfo templateInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
+ private boolean usingBackendSnapshotFor(SnapshotInfo snapshotInfo) {
+ String property = getProperty(snapshotInfo.getId(), "takeSnapshot");
+
+ return Boolean.parseBoolean(property);
+ }
+
+ private void handleCreateTemplateFromSnapshot(SnapshotInfo snapshotInfo, TemplateInfo templateInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
try {
snapshotInfo.processEvent(Event.CopyingRequested);
}
catch (Exception ex) {
throw new CloudRuntimeException("This snapshot is not currently in a state where it can be used to create a template.");
}
- HostVO hostVO = getHost(snapshotInfo.getDataStore().getId());
- DataStore srcDataStore = snapshotInfo.getDataStore();
-
- String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
- int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
- CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), templateInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+ HostVO hostVO = getHost(snapshotInfo);
- String errMsg = null;
+ boolean usingBackendSnapshot = usingBackendSnapshotFor(snapshotInfo);
+ boolean computeClusterSupportsResign = computeClusterSupportsResign(hostVO.getClusterId());
- CopyCmdAnswer copyCmdAnswer = null;
+ if (usingBackendSnapshot && !computeClusterSupportsResign) {
+ throw new CloudRuntimeException("Unable to locate an applicable host with which to perform a resignature operation");
+ }
try {
- _volumeService.grantAccess(snapshotInfo, hostVO, srcDataStore);
+ if (usingBackendSnapshot) {
+ createVolumeFromSnapshot(hostVO, snapshotInfo, true);
+ }
- Map<String, String> srcDetails = getSnapshotDetails(_storagePoolDao.findById(srcDataStore.getId()), snapshotInfo);
+ DataStore srcDataStore = snapshotInfo.getDataStore();
- copyCommand.setOptions(srcDetails);
+ String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
+ int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
+ CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), templateInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+
+ String errMsg = null;
+
+ CopyCmdAnswer copyCmdAnswer = null;
- copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
- }
- catch (Exception ex) {
- throw new CloudRuntimeException(ex.getMessage());
- }
- finally {
try {
- _volumeService.revokeAccess(snapshotInfo, hostVO, srcDataStore);
+ // If we are using a back-end snapshot, then we should still have access to it from the hosts in the cluster that hostVO is in
+ // (because we passed in true as the third parameter to createVolumeFromSnapshot above).
+ if (usingBackendSnapshot == false) {
+ _volumeService.grantAccess(snapshotInfo, hostVO, srcDataStore);
+ }
+
+ Map<String, String> srcDetails = getSnapshotDetails(snapshotInfo);
+
+ copyCommand.setOptions(srcDetails);
+
+ copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
}
catch (Exception ex) {
- s_logger.debug(ex.getMessage(), ex);
+ throw new CloudRuntimeException(ex.getMessage());
}
+ finally {
+ try {
+ _volumeService.revokeAccess(snapshotInfo, hostVO, srcDataStore);
+ }
+ catch (Exception ex) {
+ s_logger.debug(ex.getMessage(), ex);
+ }
- if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
- if (copyCmdAnswer != null && copyCmdAnswer.getDetails() != null && !copyCmdAnswer.getDetails().isEmpty()) {
- errMsg = copyCmdAnswer.getDetails();
+ if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
+ if (copyCmdAnswer != null && !StringUtils.isEmpty(copyCmdAnswer.getDetails())) {
+ errMsg = copyCmdAnswer.getDetails();
+ }
+ else {
+ errMsg = "Unable to perform host-side operation";
+ }
}
- else {
- errMsg = "Unable to perform host-side operation";
+
+ try {
+ if (errMsg == null) {
+ snapshotInfo.processEvent(Event.OperationSuccessed);
+ }
+ else {
+ snapshotInfo.processEvent(Event.OperationFailed);
+ }
+ }
+ catch (Exception ex) {
+ s_logger.debug(ex.getMessage(), ex);
}
}
- try {
- if (errMsg == null) {
- snapshotInfo.processEvent(Event.OperationSuccessed);
+ CopyCommandResult result = new CopyCommandResult(null, copyCmdAnswer);
+
+ result.setResult(errMsg);
+
+ callback.complete(result);
+ }
+ finally {
+ if (usingBackendSnapshot) {
+ deleteVolumeFromSnapshot(snapshotInfo);
+ }
+ }
+ }
+
+ /**
+ * Clones a template present on the storage to a new volume and resignatures it.
+ *
+ * @param srcData source template
+ * @param destData destination ROOT volume
+ * @param callback for async
+ */
+ private void handleCreateVolumeFromTemplateBothOnStorageSystem(DataObject srcData, DataObject destData, AsyncCompletionCallback<CopyCommandResult> callback) {
+ CopyCmdAnswer copyCmdAnswer = null;
+ String errMsg = null;
+
+ TemplateInfo templateInfo = (TemplateInfo)srcData;
+ VolumeInfo volumeInfo = (VolumeInfo)destData;
--- End diff --
guess the Preconditions.checkArguments above handles this, does it @jburwell ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r59081457
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -361,59 +608,225 @@ private String getProperty(long snapshotId, String property) {
}
private Map<String, String> getVolumeDetails(VolumeInfo volumeInfo) {
- Map<String, String> sourceDetails = new HashMap<String, String>();
+ Map<String, String> volumeDetails = new HashMap<String, String>();
VolumeVO volumeVO = _volumeDao.findById(volumeInfo.getId());
long storagePoolId = volumeVO.getPoolId();
StoragePoolVO storagePoolVO = _storagePoolDao.findById(storagePoolId);
- sourceDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
- sourceDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
- sourceDetails.put(DiskTO.IQN, volumeVO.get_iScsiName());
+ volumeDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
+ volumeDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
+ volumeDetails.put(DiskTO.IQN, volumeVO.get_iScsiName());
ChapInfo chapInfo = _volumeService.getChapInfo(volumeInfo, volumeInfo.getDataStore());
if (chapInfo != null) {
- sourceDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, chapInfo.getInitiatorUsername());
- sourceDetails.put(DiskTO.CHAP_INITIATOR_SECRET, chapInfo.getInitiatorSecret());
- sourceDetails.put(DiskTO.CHAP_TARGET_USERNAME, chapInfo.getTargetUsername());
- sourceDetails.put(DiskTO.CHAP_TARGET_SECRET, chapInfo.getTargetSecret());
+ volumeDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, chapInfo.getInitiatorUsername());
+ volumeDetails.put(DiskTO.CHAP_INITIATOR_SECRET, chapInfo.getInitiatorSecret());
+ volumeDetails.put(DiskTO.CHAP_TARGET_USERNAME, chapInfo.getTargetUsername());
+ volumeDetails.put(DiskTO.CHAP_TARGET_SECRET, chapInfo.getTargetSecret());
+ }
+
+ return volumeDetails;
+ }
+
+ private Map<String, String> getSnapshotDetails(SnapshotInfo snapshotInfo) {
+ Map<String, String> snapshotDetails = new HashMap<String, String>();
+
+ long storagePoolId = snapshotInfo.getDataStore().getId();
+ StoragePoolVO storagePoolVO = _storagePoolDao.findById(storagePoolId);
+
+ snapshotDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
+ snapshotDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
+
+ long snapshotId = snapshotInfo.getId();
+
+ snapshotDetails.put(DiskTO.IQN, getProperty(snapshotId, DiskTO.IQN));
+
+ snapshotDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, getProperty(snapshotId, DiskTO.CHAP_INITIATOR_USERNAME));
+ snapshotDetails.put(DiskTO.CHAP_INITIATOR_SECRET, getProperty(snapshotId, DiskTO.CHAP_INITIATOR_SECRET));
+ snapshotDetails.put(DiskTO.CHAP_TARGET_USERNAME, getProperty(snapshotId, DiskTO.CHAP_TARGET_USERNAME));
+ snapshotDetails.put(DiskTO.CHAP_TARGET_SECRET, getProperty(snapshotId, DiskTO.CHAP_TARGET_SECRET));
+
+ return snapshotDetails;
+ }
+
+ private HostVO getHost(SnapshotInfo snapshotInfo) {
+ HostVO hostVO = getHost(snapshotInfo.getDataCenterId(), true);
+
+ if (hostVO == null) {
+ hostVO = getHost(snapshotInfo.getDataCenterId(), false);
+
+ if (hostVO == null) {
+ throw new CloudRuntimeException("Unable to locate an applicable host");
+ }
}
- return sourceDetails;
+ return hostVO;
}
- public HostVO getHost(long dataStoreId) {
- StoragePoolVO storagePoolVO = _storagePoolDao.findById(dataStoreId);
+ private HostVO getHost(Long zoneId, boolean computeClusterMustSupportResign) {
+ if (zoneId == null) {
+ throw new CloudRuntimeException("Zone ID cannot be null.");
+ }
- List<? extends Cluster> clusters = _mgr.searchForClusters(storagePoolVO.getDataCenterId(), new Long(0), Long.MAX_VALUE, HypervisorType.XenServer.toString());
+ List<? extends Cluster> clusters = _mgr.searchForClusters(zoneId, new Long(0), Long.MAX_VALUE, HypervisorType.XenServer.toString());
if (clusters == null) {
- throw new CloudRuntimeException("Unable to locate an applicable cluster");
+ clusters = new ArrayList<>();
}
+ Collections.shuffle(clusters, new Random(System.nanoTime()));
+
+ clusters:
for (Cluster cluster : clusters) {
if (cluster.getAllocationState() == AllocationState.Enabled) {
List<HostVO> hosts = _hostDao.findByClusterId(cluster.getId());
if (hosts != null) {
+ Collections.shuffle(hosts, new Random(System.nanoTime()));
+
for (HostVO host : hosts) {
if (host.getResourceState() == ResourceState.Enabled) {
- return host;
+ if (computeClusterMustSupportResign) {
+ if (computeClusterSupportsResign(cluster.getId())) {
+ return host;
+ }
+ else {
+ // no other host in the cluster in question should be able to satisfy our requirements here, so move on to the next cluster
+ continue clusters;
+ }
+ }
+ else {
+ return host;
+ }
}
}
}
}
}
- throw new CloudRuntimeException("Unable to locate an applicable cluster");
+ return null;
}
@Override
public void copyAsync(Map<VolumeInfo, DataStore> volumeMap, VirtualMachineTO vmTo, Host srcHost, Host destHost, AsyncCompletionCallback<CopyCommandResult> callback) {
CopyCommandResult result = new CopyCommandResult(null, null);
+
result.setResult("Unsupported operation requested for copying data.");
+
callback.complete(result);
}
+
+ private Map<String, String> getDetails(DataObject dataObj) {
+ if (dataObj instanceof VolumeInfo) {
+ return getVolumeDetails((VolumeInfo)dataObj);
+ }
+ else if (dataObj instanceof SnapshotInfo) {
+ return getSnapshotDetails((SnapshotInfo)dataObj);
+ }
+
+ throw new CloudRuntimeException("'dataObj' must be of type 'VolumeInfo' or 'SnapshotInfo'.");
+ }
+
+ private CopyCmdAnswer performResignature(DataObject dataObj, HostVO hostVO) {
+ return performResignature(dataObj, hostVO, false);
+ }
+
+ private CopyCmdAnswer performResignature(DataObject dataObj, HostVO hostVO, boolean keepGrantedAccess) {
+ long storagePoolId = dataObj.getDataStore().getId();
+ DataStore dataStore = _dataStoreMgr.getDataStore(storagePoolId, DataStoreRole.Primary);
+
+ Map<String, String> details = getDetails(dataObj);
+
+ ResignatureCommand command = new ResignatureCommand(details);
+
+ ResignatureAnswer answer = null;
+
+ try {
+ _volumeService.grantAccess(dataObj, hostVO, dataStore);
+
+ answer = (ResignatureAnswer)_agentMgr.send(hostVO.getId(), command);
+ }
+ catch (Exception ex) {
+ keepGrantedAccess = false;
+
+ throw new CloudRuntimeException(ex.getMessage());
+ }
+ finally {
+ if (keepGrantedAccess == false) {
+ try {
+ _volumeService.revokeAccess(dataObj, hostVO, dataStore);
+ }
+ catch (Exception ex) {
+ s_logger.debug(ex.getMessage(), ex);
--- End diff --
1. Why not log this ignored exception to ``WARN``? It seems like the operator would want to know if we failed to rollback the operation.
1. Please add some contextual information around the exception message to explain that we failed to rollback access to the datastore on the host.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r63110957
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -132,31 +171,52 @@ public void copyAsync(DataObject srcData, DataObject destData, Host destHost, As
validate(snapshotInfo);
- boolean canHandleSrc = canHandle(srcData.getDataStore());
+ boolean canHandleSrc = canHandle(srcData);
if (canHandleSrc && destData instanceof TemplateInfo &&
(destData.getDataStore().getRole() == DataStoreRole.Image || destData.getDataStore().getRole() == DataStoreRole.ImageCache)) {
handleCreateTemplateFromSnapshot(snapshotInfo, (TemplateInfo)destData, callback);
+
return;
}
if (destData instanceof VolumeInfo) {
VolumeInfo volumeInfo = (VolumeInfo)destData;
- boolean canHandleDest = canHandle(destData.getDataStore());
+
+ boolean canHandleDest = canHandle(destData);
if (canHandleSrc && canHandleDest) {
- handleCreateVolumeFromSnapshotBothOnStorageSystem(snapshotInfo, volumeInfo, callback);
- return;
+ if (snapshotInfo.getDataStore().getId() == volumeInfo.getDataStore().getId()) {
+ handleCreateVolumeFromSnapshotBothOnStorageSystem(snapshotInfo, volumeInfo, callback);
+ return;
+ }
+ else {
+ throw new UnsupportedOperationException("This operation is not supported (DataStoreCapabilities.STORAGE_SYSTEM_SNAPSHOT " +
+ "not supported by source or destination storage plug-in).");
+ }
}
+
if (canHandleSrc) {
throw new UnsupportedOperationException("This operation is not supported (DataStoreCapabilities.STORAGE_SYSTEM_SNAPSHOT " +
"not supported by destination storage plug-in).");
}
+
if (canHandleDest) {
throw new UnsupportedOperationException("This operation is not supported (DataStoreCapabilities.STORAGE_SYSTEM_SNAPSHOT " +
"not supported by source storage plug-in).");
}
}
+ } else if (srcData instanceof TemplateInfo && destData instanceof VolumeInfo) {
+ boolean canHandleSrc = canHandle(srcData);
+
+ if (!canHandleSrc) {
+ throw new UnsupportedOperationException("This operation is not supported (DataStoreCapabilities.STORAGE_CAN_CREATE_VOLUME_FROM_VOLUME " +
--- End diff --
@mike-tutkowski is it possible to add this information to the log message?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by mike-tutkowski <gi...@git.apache.org>.
Github user mike-tutkowski commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r63131310
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -132,31 +171,52 @@ public void copyAsync(DataObject srcData, DataObject destData, Host destHost, As
validate(snapshotInfo);
- boolean canHandleSrc = canHandle(srcData.getDataStore());
+ boolean canHandleSrc = canHandle(srcData);
if (canHandleSrc && destData instanceof TemplateInfo &&
(destData.getDataStore().getRole() == DataStoreRole.Image || destData.getDataStore().getRole() == DataStoreRole.ImageCache)) {
handleCreateTemplateFromSnapshot(snapshotInfo, (TemplateInfo)destData, callback);
+
return;
}
if (destData instanceof VolumeInfo) {
VolumeInfo volumeInfo = (VolumeInfo)destData;
- boolean canHandleDest = canHandle(destData.getDataStore());
+
+ boolean canHandleDest = canHandle(destData);
if (canHandleSrc && canHandleDest) {
- handleCreateVolumeFromSnapshotBothOnStorageSystem(snapshotInfo, volumeInfo, callback);
- return;
+ if (snapshotInfo.getDataStore().getId() == volumeInfo.getDataStore().getId()) {
+ handleCreateVolumeFromSnapshotBothOnStorageSystem(snapshotInfo, volumeInfo, callback);
+ return;
+ }
+ else {
+ throw new UnsupportedOperationException("This operation is not supported (DataStoreCapabilities.STORAGE_SYSTEM_SNAPSHOT " +
--- End diff --
Done
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by mike-tutkowski <gi...@git.apache.org>.
Github user mike-tutkowski commented on the pull request:
https://github.com/apache/cloudstack/pull/1403#issuecomment-207543372
No problem, @swill, I can just squash them all. I was just thinking that the fact that we are merging can preserve the individual history better (in a different branch that makes its way back to the master branch), but I'm fine with just dropping the history and having a single commit.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on the pull request:
https://github.com/apache/cloudstack/pull/1403#issuecomment-208421003
@mike-tutkowski I will re-review the patch later today (11 April 2015).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r59083160
--- Diff: engine/storage/integration-test/test/org/apache/cloudstack/storage/test/FakePrimaryDataStoreDriver.java ---
@@ -65,8 +66,13 @@ public long getUsedIops(StoragePool storagePool) {
}
@Override
- public long getVolumeSizeIncludingHypervisorSnapshotReserve(Volume volume, StoragePool pool) {
- return volume.getSize();
+ public long getDataObjectSizeIncludingHypervisorSnapshotReserve(DataObject dataObject, StoragePool pool) {
+ return dataObject.getSize();
+ }
+
+ @Override
+ public long getBytesRequiredForTemplate(TemplateInfo templateInfo, StoragePool storagePool) {
+ return 0;
--- End diff --
Minor Nit: ``0`` is an integer value. ``0L`` is a long value. Java automatically widens the value, but it is more hygienic to match the type for a constant value.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
Re: [GitHub] cloudstack pull request: Taking fast and efficient
volume snapshot...
Posted by Mike Tutkowski <mi...@solidfire.com>.
None of those individual commits should break CI.
On Tuesday, February 16, 2016, DaanHoogland <gi...@git.apache.org> wrote:
> Github user DaanHoogland commented on a diff in the pull request:
>
> https://github.com/apache/cloudstack/pull/1403#discussion_r53058692
>
> --- Diff:
> engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java
> ---
> @@ -172,6 +195,12 @@ private void validate(SnapshotInfo snapshotInfo) {
> }
> }
>
> + private boolean usingBackendSnapshotFor(SnapshotInfo
> snapshotInfo) {
> + String property = getProperty(snapshotInfo.getId(),
> "takeSnapshot");
> +
> + return Boolean.parseBoolean(property);
> + }
> +
> private Void handleCreateTemplateFromSnapshot(SnapshotInfo
> snapshotInfo, TemplateInfo templateInfo,
> AsyncCompletionCallback<CopyCommandResult> callback) {
> --- End diff --
>
> @mike-tutkowski if commits are atomic and not breaking ci, yes, else,
> no. Please use your own judgement.
>
>
> ---
> If your project is set up for it, you can reply to this email and have your
> reply appear on GitHub as well. If your project does not have this feature
> enabled and wishes so, or if the feature is enabled but not working, please
> contact infrastructure at infrastructure@apache.org <javascript:;> or
> file a JIRA ticket
> with INFRA.
> ---
>
--
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkowski@solidfire.com
o: 303.746.7302
Advancing the way the world uses the cloud
<http://solidfire.com/solution/overview/?video=play>*™*
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by DaanHoogland <gi...@git.apache.org>.
Github user DaanHoogland commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r53058692
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -172,6 +195,12 @@ private void validate(SnapshotInfo snapshotInfo) {
}
}
+ private boolean usingBackendSnapshotFor(SnapshotInfo snapshotInfo) {
+ String property = getProperty(snapshotInfo.getId(), "takeSnapshot");
+
+ return Boolean.parseBoolean(property);
+ }
+
private Void handleCreateTemplateFromSnapshot(SnapshotInfo snapshotInfo, TemplateInfo templateInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
--- End diff --
@mike-tutkowski if commits are atomic and not breaking ci, yes, else, no. Please use your own judgement.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r59085610
--- Diff: engine/storage/snapshot/src/org/apache/cloudstack/storage/snapshot/XenserverSnapshotStrategy.java ---
@@ -279,7 +284,52 @@ public boolean deleteSnapshot(Long snapshotId) {
@Override
public boolean revertSnapshot(SnapshotInfo snapshot) {
- throw new CloudRuntimeException("revert Snapshot is not supported");
+ if (canHandle(snapshot,SnapshotOperation.REVERT) == StrategyPriority.CANT_HANDLE) {
+ throw new UnsupportedOperationException("Reverting not supported. Create a template or volume based on the snapshot instead.");
+ }
+
+ SnapshotVO snapshotVO = _snapshotDao.acquireInLockTable(snapshot.getId());
+
+ if (snapshotVO == null) {
+ throw new CloudRuntimeException("Failed to get lock on snapshot:" + snapshot.getId());
+ }
+
+ try {
+ VolumeInfo volumeInfo = snapshot.getBaseVolume();
+ StoragePool store = (StoragePool)volumeInfo.getDataStore();
+
+ if (store != null && store.getStatus() != StoragePoolStatus.Up) {
+ snapshot.processEvent(Event.OperationFailed);
+
+ throw new CloudRuntimeException("store is not in up state");
+ }
+
+ volumeInfo.stateTransit(Volume.Event.RevertSnapshotRequested);
+
+ boolean result = false;
+
+ try {
+ result = snapshotSvr.revertSnapshot(snapshot);
+
+ if (!result) {
+ s_logger.debug("Failed to revert snapshot: " + snapshot.getId());
--- End diff --
Why log an error here when the exception will eventually logged?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r59073600
--- Diff: engine/orchestration/src/org/apache/cloudstack/engine/orchestration/VolumeOrchestrator.java ---
@@ -394,6 +394,7 @@ public VolumeInfo createVolumeFromSnapshot(Volume volume, Snapshot snapshot, Use
}
// create volume on primary from snapshot
+ /* TODO: Change HERE!!! */
--- End diff --
What does this TODO refer to? What work is outstanding?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r59082769
--- Diff: engine/storage/image/src/org/apache/cloudstack/storage/image/store/TemplateObject.java ---
@@ -189,12 +189,24 @@ public void processEvent(ObjectInDataStoreStateMachine.Event event, Answer answe
TemplateObjectTO newTemplate = (TemplateObjectTO)cpyAnswer.getNewData();
VMTemplateStoragePoolVO templatePoolRef = templatePoolDao.findByPoolTemplate(getDataStore().getId(), getId());
templatePoolRef.setDownloadPercent(100);
- if (newTemplate.getSize() != null) {
+
+ // In the case of managed storage, the template size may already be specified (by the storage plug-in), so do not overwrite it.
+ if (templatePoolRef.getTemplateSize() == 0 && newTemplate.getSize() != null) {
templatePoolRef.setTemplateSize(newTemplate.getSize());
}
+
templatePoolRef.setDownloadState(Status.DOWNLOADED);
- templatePoolRef.setLocalDownloadPath(newTemplate.getPath());
- templatePoolRef.setInstallPath(newTemplate.getPath());
+
+ // In the case of managed storage, the local download path may already be specified (by the storage plug-in), so do not overwrite it.
+ if (templatePoolRef.getLocalDownloadPath() == null) {
--- End diff --
Consider using ``Strings.isNullorEmpty`` to cover both the ``null`` and empty cases.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by rodrigo93 <gi...@git.apache.org>.
Github user rodrigo93 commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r52845493
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -172,6 +195,12 @@ private void validate(SnapshotInfo snapshotInfo) {
}
}
+ private boolean usingBackendSnapshotFor(SnapshotInfo snapshotInfo) {
+ String property = getProperty(snapshotInfo.getId(), "takeSnapshot");
+
+ return Boolean.parseBoolean(property);
+ }
+
private Void handleCreateTemplateFromSnapshot(SnapshotInfo snapshotInfo, TemplateInfo templateInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
--- End diff --
Hi @mike-tutkowski ,
Consider changing "Void" to "void" and removing the "return null;" at the end of this method.
Another suggestion is to create a single method, if possible, for the lines 260 to 266 and 363 to 371, this would avoid some duplication of code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by mike-tutkowski <gi...@git.apache.org>.
Github user mike-tutkowski commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r63083633
--- Diff: engine/storage/snapshot/src/org/apache/cloudstack/storage/snapshot/XenserverSnapshotStrategy.java ---
@@ -289,7 +291,7 @@ public boolean deleteSnapshot(Long snapshotId) {
@Override
public boolean revertSnapshot(SnapshotInfo snapshot) {
if (canHandle(snapshot,SnapshotOperation.REVERT) == StrategyPriority.CANT_HANDLE) {
- throw new UnsupportedOperationException("Reverting not supported. Create a template or volume based on the snapshot instead.");
+ throw new CloudRuntimeException("Reverting not supported. Create a template or volume based on the snapshot instead.");
--- End diff --
@DaanHoogland It was per John's suggestion.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by swill <gi...@git.apache.org>.
Github user swill commented on the pull request:
https://github.com/apache/cloudstack/pull/1403#issuecomment-219702269
this one is ready...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by DaanHoogland <gi...@git.apache.org>.
Github user DaanHoogland commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r62876182
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -361,59 +603,221 @@ private String getProperty(long snapshotId, String property) {
}
private Map<String, String> getVolumeDetails(VolumeInfo volumeInfo) {
- Map<String, String> sourceDetails = new HashMap<String, String>();
+ Map<String, String> volumeDetails = new HashMap<String, String>();
VolumeVO volumeVO = _volumeDao.findById(volumeInfo.getId());
long storagePoolId = volumeVO.getPoolId();
StoragePoolVO storagePoolVO = _storagePoolDao.findById(storagePoolId);
- sourceDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
- sourceDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
- sourceDetails.put(DiskTO.IQN, volumeVO.get_iScsiName());
+ volumeDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
+ volumeDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
+ volumeDetails.put(DiskTO.IQN, volumeVO.get_iScsiName());
ChapInfo chapInfo = _volumeService.getChapInfo(volumeInfo, volumeInfo.getDataStore());
if (chapInfo != null) {
- sourceDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, chapInfo.getInitiatorUsername());
- sourceDetails.put(DiskTO.CHAP_INITIATOR_SECRET, chapInfo.getInitiatorSecret());
- sourceDetails.put(DiskTO.CHAP_TARGET_USERNAME, chapInfo.getTargetUsername());
- sourceDetails.put(DiskTO.CHAP_TARGET_SECRET, chapInfo.getTargetSecret());
+ volumeDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, chapInfo.getInitiatorUsername());
+ volumeDetails.put(DiskTO.CHAP_INITIATOR_SECRET, chapInfo.getInitiatorSecret());
+ volumeDetails.put(DiskTO.CHAP_TARGET_USERNAME, chapInfo.getTargetUsername());
+ volumeDetails.put(DiskTO.CHAP_TARGET_SECRET, chapInfo.getTargetSecret());
}
- return sourceDetails;
+ return volumeDetails;
+ }
+
+ private Map<String, String> getSnapshotDetails(SnapshotInfo snapshotInfo) {
+ Map<String, String> snapshotDetails = new HashMap<String, String>();
+
+ long storagePoolId = snapshotInfo.getDataStore().getId();
+ StoragePoolVO storagePoolVO = _storagePoolDao.findById(storagePoolId);
+
+ snapshotDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
+ snapshotDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
+
+ long snapshotId = snapshotInfo.getId();
+
+ snapshotDetails.put(DiskTO.IQN, getProperty(snapshotId, DiskTO.IQN));
+
+ snapshotDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, getProperty(snapshotId, DiskTO.CHAP_INITIATOR_USERNAME));
+ snapshotDetails.put(DiskTO.CHAP_INITIATOR_SECRET, getProperty(snapshotId, DiskTO.CHAP_INITIATOR_SECRET));
+ snapshotDetails.put(DiskTO.CHAP_TARGET_USERNAME, getProperty(snapshotId, DiskTO.CHAP_TARGET_USERNAME));
+ snapshotDetails.put(DiskTO.CHAP_TARGET_SECRET, getProperty(snapshotId, DiskTO.CHAP_TARGET_SECRET));
+
+ return snapshotDetails;
}
- public HostVO getHost(long dataStoreId) {
- StoragePoolVO storagePoolVO = _storagePoolDao.findById(dataStoreId);
+ private HostVO getHost(SnapshotInfo snapshotInfo) {
+ HostVO hostVO = getHost(snapshotInfo.getDataCenterId(), true);
- List<? extends Cluster> clusters = _mgr.searchForClusters(storagePoolVO.getDataCenterId(), new Long(0), Long.MAX_VALUE, HypervisorType.XenServer.toString());
+ if (hostVO == null) {
+ hostVO = getHost(snapshotInfo.getDataCenterId(), false);
- if (clusters == null) {
- throw new CloudRuntimeException("Unable to locate an applicable cluster");
+ if (hostVO == null) {
+ throw new CloudRuntimeException("Unable to locate an applicable host in data center with ID = " + snapshotInfo.getDataCenterId());
+ }
}
- for (Cluster cluster : clusters) {
- if (cluster.getAllocationState() == AllocationState.Enabled) {
- List<HostVO> hosts = _hostDao.findByClusterId(cluster.getId());
+ return hostVO;
+ }
- if (hosts != null) {
- for (HostVO host : hosts) {
- if (host.getResourceState() == ResourceState.Enabled) {
- return host;
- }
- }
+ private HostVO getHost(Long zoneId, boolean computeClusterMustSupportResign) {
+ Preconditions.checkArgument(zoneId != null, "Zone ID cannot be null.");
+
+ List<HostVO> hosts = _hostDao.listByDataCenterIdAndHypervisorType(zoneId, HypervisorType.XenServer);
+
+ if (hosts == null) {
+ return null;
+ }
+
+ List<Long> clustersToSkip = new ArrayList<>();
+
+ Collections.shuffle(hosts, random);
+
+ for (HostVO host : hosts) {
+ if (computeClusterMustSupportResign) {
+ long clusterId = host.getClusterId();
+
+ if (clustersToSkip.contains(clusterId)) {
+ continue;
+ }
+
+ if (_clusterDao.computeClusterSupportsResign(clusterId)) {
+ return host;
+ }
+ else {
+ clustersToSkip.add(clusterId);
}
}
+ else {
+ return host;
+ }
}
- throw new CloudRuntimeException("Unable to locate an applicable cluster");
+ return null;
}
@Override
public void copyAsync(Map<VolumeInfo, DataStore> volumeMap, VirtualMachineTO vmTo, Host srcHost, Host destHost, AsyncCompletionCallback<CopyCommandResult> callback) {
CopyCommandResult result = new CopyCommandResult(null, null);
+
result.setResult("Unsupported operation requested for copying data.");
+
callback.complete(result);
}
+
+ private Map<String, String> getDetails(DataObject dataObj) {
+ if (dataObj instanceof VolumeInfo) {
+ return getVolumeDetails((VolumeInfo)dataObj);
+ }
+ else if (dataObj instanceof SnapshotInfo) {
+ return getSnapshotDetails((SnapshotInfo)dataObj);
+ }
+
+ throw new CloudRuntimeException("'dataObj' must be of type 'VolumeInfo' or 'SnapshotInfo'.");
+ }
+
+ private CopyCmdAnswer performResignature(DataObject dataObj, HostVO hostVO) {
--- End diff --
performResigning seems a better name, @mike-tutkowski ???
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by mike-tutkowski <gi...@git.apache.org>.
Github user mike-tutkowski commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r63086591
--- Diff: engine/storage/volume/src/org/apache/cloudstack/storage/volume/VolumeServiceImpl.java ---
@@ -554,6 +574,51 @@ protected Void managedCopyBaseImageCallback(AsyncCallbackDispatcher<VolumeServic
return null;
}
+ protected Void createManagedTemplateImageCallback(AsyncCallbackDispatcher<VolumeServiceImpl, CreateCmdResult> callback, CreateVolumeContext<CreateCmdResult> context) {
+ CreateCmdResult result = callback.getResult();
+ VolumeApiResult res = new VolumeApiResult(null);
+
+ res.setResult(result.getResult());
+
+ AsyncCallFuture<VolumeApiResult> future = context.getFuture();
+ DataObject templateOnPrimaryStoreObj = context.getVolume();
+
+ if (result.isSuccess()) {
+ ((TemplateObject)templateOnPrimaryStoreObj).setInstallPath(result.getPath());
+ templateOnPrimaryStoreObj.processEvent(Event.OperationSuccessed, result.getAnswer());
+ }
+ else {
+ templateOnPrimaryStoreObj.processEvent(Event.OperationFailed);
+ }
+
+ future.complete(res);
+
+ return null;
+ }
+
+ protected Void copyManagedTemplateCallback(AsyncCallbackDispatcher<VolumeServiceImpl, CopyCommandResult> callback, CreateBaseImageContext<VolumeApiResult> context) {
--- End diff --
@DaanHoogland I changed them back to the more traditional "void" return type. I'm not sure why "Void" is being used for those kinds of methods in this class (I was thinking maybe for AOP).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by mike-tutkowski <gi...@git.apache.org>.
Github user mike-tutkowski commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r63082200
--- Diff: engine/schema/src/com/cloud/dc/dao/ClusterDaoImpl.java ---
@@ -260,4 +268,41 @@ public boolean remove(Long id) {
sc.setParameters("dataCenterId", zoneId);
return customSearch(sc, null);
}
+
+ @Override
+ public boolean computeClusterSupportsResign(long clusterId) {
--- End diff --
@DaanHoogland Done
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by mike-tutkowski <gi...@git.apache.org>.
Github user mike-tutkowski commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r63139458
--- Diff: engine/schema/src/com/cloud/dc/dao/ClusterDaoImpl.java ---
@@ -260,4 +268,41 @@ public boolean remove(Long id) {
sc.setParameters("dataCenterId", zoneId);
return customSearch(sc, null);
}
+
+ @Override
+ public boolean computeWhetherClusterSupportsResigning(long clusterId) {
+ ClusterVO cluster = findById(clusterId);
+
+ if (cluster == null || cluster.getAllocationState() != Grouping.AllocationState.Enabled) {
+ return false;
+ }
+
+ List<HostVO> hosts = hostDao.findByClusterId(clusterId);
+
+ if (hosts == null) {
+ return false;
+ }
+
+ for (HostVO host : hosts) {
+ if (host == null) {
+ return false;
+ }
+
+ DetailVO hostDetail = hostDetailsDao.findDetail(host.getId(), "supportsResign");
+
+ if (hostDetail == null) {
+ return false;
+ }
+
+ String value = hostDetail.getValue();
+
+ Boolean booleanValue = Boolean.valueOf(value);
+
+ if (booleanValue == false) {
+ return false;
+ }
+ }
+
+ return true;
+ }
--- End diff --
Actually, I think we just need to join the host and the host_details tables (because cluster ID should be in the host table).
We'd want all of the hosts with a particular cluster ID selected from the host table and then we'd want to join on host ID in the host_details table and further refine the returned hosts by those that have a "supportsResign" value in the name field.
Do you happen to know of any code of ours that allows you to SQL filter the returned values based on values that exist in not just one, but two tables? From what I've seen, we seem to be pretty heavily filtering only on data present in one table (even when we join with another table).
If this were just raw SQL, it would be pretty easy, but we've got that Java DB layer that this needs to fit within.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by mike-tutkowski <gi...@git.apache.org>.
Github user mike-tutkowski commented on the pull request:
https://github.com/apache/cloudstack/pull/1403#issuecomment-210087782
No problem. :)
I had forgotten that I had renamed that file at one point. So, when I saw that you had a file by that name, I was really confused how you would have had access to a file that I had not yet checked into the repo (the current file with that name is for a project I'm still developing and I haven't opened a PR on it yet).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by mike-tutkowski <gi...@git.apache.org>.
Github user mike-tutkowski commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r52114577
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -180,70 +208,119 @@ private Void handleCreateTemplateFromSnapshot(SnapshotInfo snapshotInfo, Templat
throw new CloudRuntimeException("This snapshot is not currently in a state where it can be used to create a template.");
}
- HostVO hostVO = getHost(snapshotInfo.getDataStore().getId());
- DataStore srcDataStore = snapshotInfo.getDataStore();
-
- String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
- int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
- CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), templateInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+ HostVO hostVO = getXenServerHost(snapshotInfo);
- String errMsg = null;
+ boolean usingBackendSnapshot = usingBackendSnapshotFor(snapshotInfo);
+ boolean computeClusterSupportsResign = computeClusterSupportsResign(hostVO.getClusterId());
- CopyCmdAnswer copyCmdAnswer = null;
+ if (usingBackendSnapshot && !computeClusterSupportsResign) {
+ throw new CloudRuntimeException("Unable to locate an applicable host with which to perform a resignature operation");
+ }
try {
- _volumeService.grantAccess(snapshotInfo, hostVO, srcDataStore);
+ if (usingBackendSnapshot) {
+ createVolumeFromSnapshot(hostVO, snapshotInfo, true);
+ }
- Map<String, String> srcDetails = getSnapshotDetails(_storagePoolDao.findById(srcDataStore.getId()), snapshotInfo);
+ DataStore srcDataStore = snapshotInfo.getDataStore();
- copyCommand.setOptions(srcDetails);
+ String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
+ int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
+ CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), templateInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+
+ String errMsg = null;
+
+ CopyCmdAnswer copyCmdAnswer = null;
- copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
- }
- catch (Exception ex) {
- throw new CloudRuntimeException(ex.getMessage());
- }
- finally {
try {
- _volumeService.revokeAccess(snapshotInfo, hostVO, srcDataStore);
+ // If we are using a back-end snapshot, then we should still have access to it from the hosts in the cluster that hostVO is in
+ // (because we passed in true as the third parameter to createVolumeFromSnapshot above).
+ if (usingBackendSnapshot == false) {
+ _volumeService.grantAccess(snapshotInfo, hostVO, srcDataStore);
+ }
+
+ Map<String, String> srcDetails = getSnapshotDetails(snapshotInfo);
+
+ copyCommand.setOptions(srcDetails);
+
+ copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
}
catch (Exception ex) {
- s_logger.debug(ex.getMessage(), ex);
+ throw new CloudRuntimeException(ex.getMessage());
}
-
- if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
- if (copyCmdAnswer != null && copyCmdAnswer.getDetails() != null && !copyCmdAnswer.getDetails().isEmpty()) {
- errMsg = copyCmdAnswer.getDetails();
+ finally {
+ try {
+ _volumeService.revokeAccess(snapshotInfo, hostVO, srcDataStore);
}
- else {
- errMsg = "Unable to perform host-side operation";
+ catch (Exception ex) {
+ s_logger.debug(ex.getMessage(), ex);
}
- }
- try {
- if (errMsg == null) {
- snapshotInfo.processEvent(Event.OperationSuccessed);
+ if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
+ if (copyCmdAnswer != null && copyCmdAnswer.getDetails() != null && !copyCmdAnswer.getDetails().isEmpty()) {
--- End diff --
Done - thanks
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by rafaelweingartner <gi...@git.apache.org>.
Github user rafaelweingartner commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r59076866
--- Diff: core/src/com/cloud/agent/api/StartupRoutingCommand.java ---
@@ -35,7 +35,7 @@
long memory;
long dom0MinMemory;
boolean poolSync;
-
+ private boolean _supportsClonedVolumes;
--- End diff --
what about starting to use variables without "_"?
I know ACS is full of variables starting with "_", but I believe we should start going toward a better pattern.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r59077514
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -255,99 +448,153 @@ private Void handleCreateVolumeFromSnapshotBothOnStorageSystem(SnapshotInfo snap
VolumeApiResult result = future.get();
+ if (volumeDetail != null) {
+ _volumeDetailsDao.remove(volumeDetail.getId());
+ }
+
if (result.isFailed()) {
s_logger.debug("Failed to create a volume: " + result.getResult());
throw new CloudRuntimeException(result.getResult());
}
- }
- catch (Exception ex) {
- throw new CloudRuntimeException(ex.getMessage());
- }
- volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
+ volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
- volumeInfo.processEvent(Event.MigrationRequested);
+ volumeInfo.processEvent(Event.MigrationRequested);
- volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
+ volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
- HostVO hostVO = getHost(snapshotInfo.getDataStore().getId());
+ if (useCloning) {
+ copyCmdAnswer = performResignature(volumeInfo, hostVO);
+ }
+ else {
+ // asking for a XenServer host here so we don't always prefer to use XenServer hosts that support resigning
+ // even when we don't need those hosts to do this kind of copy work
+ hostVO = getHost(snapshotInfo.getDataCenterId(), false);
- String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
- int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
- CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), volumeInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+ copyCmdAnswer = performCopyOfVdi(volumeInfo, snapshotInfo, hostVO);
+ }
- CopyCmdAnswer copyCmdAnswer = null;
+ if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
+ if (copyCmdAnswer != null && !StringUtils.isEmpty(copyCmdAnswer.getDetails())) {
+ errMsg = copyCmdAnswer.getDetails();
+ }
+ else {
+ errMsg = "Unable to perform host-side operation";
+ }
+ }
+ }
+ catch (Exception ex) {
+ errMsg = ex.getMessage() != null ? ex.getMessage() : "Copy operation failed";
+ }
- try {
- _volumeService.grantAccess(snapshotInfo, hostVO, snapshotInfo.getDataStore());
- _volumeService.grantAccess(volumeInfo, hostVO, volumeInfo.getDataStore());
+ CopyCommandResult result = new CopyCommandResult(null, copyCmdAnswer);
- Map<String, String> srcDetails = getSnapshotDetails(_storagePoolDao.findById(snapshotInfo.getDataStore().getId()), snapshotInfo);
+ result.setResult(errMsg);
- copyCommand.setOptions(srcDetails);
+ callback.complete(result);
+ }
- Map<String, String> destDetails = getVolumeDetails(volumeInfo);
+ /**
+ * If the underlying storage system is making use of read-only snapshots, this gives the storage system the opportunity to
+ * create a volume from the snapshot so that we can copy the VHD file that should be inside of the snapshot to secondary storage.
+ *
+ * The resultant volume must be writable because we need to resign the SR and the VDI that should be inside of it before we copy
+ * the VHD file to secondary storage.
+ *
+ * If the storage system is using writable snapshots, then nothing need be done by that storage system here because we can just
+ * resign the SR and the VDI that should be inside of the snapshot before copying the VHD file to secondary storage.
+ */
+ private void createVolumeFromSnapshot(HostVO hostVO, SnapshotInfo snapshotInfo, boolean keepGrantedAccess) {
+ SnapshotDetailsVO snapshotDetails = handleSnapshotDetails(snapshotInfo.getId(), "tempVolume", "create");
- copyCommand.setOptions2(destDetails);
-
- copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
- }
- catch (Exception ex) {
- throw new CloudRuntimeException(ex.getMessage());
+ try {
+ snapshotInfo.getDataStore().getDriver().createAsync(snapshotInfo.getDataStore(), snapshotInfo, null);
}
finally {
- try {
- _volumeService.revokeAccess(snapshotInfo, hostVO, snapshotInfo.getDataStore());
- }
- catch (Exception ex) {
- s_logger.debug(ex.getMessage(), ex);
- }
-
- try {
- _volumeService.revokeAccess(volumeInfo, hostVO, volumeInfo.getDataStore());
- }
- catch (Exception ex) {
- s_logger.debug(ex.getMessage(), ex);
- }
+ _snapshotDetailsDao.remove(snapshotDetails.getId());
}
- String errMsg = null;
+ CopyCmdAnswer copyCmdAnswer = performResignature(snapshotInfo, hostVO, keepGrantedAccess);
if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
- if (copyCmdAnswer != null && copyCmdAnswer.getDetails() != null && !copyCmdAnswer.getDetails().isEmpty()) {
- errMsg = copyCmdAnswer.getDetails();
+ if (copyCmdAnswer != null && !StringUtils.isEmpty(copyCmdAnswer.getDetails())) {
+ throw new CloudRuntimeException(copyCmdAnswer.getDetails());
}
else {
- errMsg = "Unable to perform host-side operation";
+ throw new CloudRuntimeException("Unable to perform host-side operation");
}
}
+ }
- CopyCommandResult result = new CopyCommandResult(null, copyCmdAnswer);
+ /**
+ * If the underlying storage system needed to create a volume from a snapshot for createVolumeFromSnapshot(HostVO, SnapshotInfo), then
+ * this is its opportunity to delete that temporary volume and restore properties in snapshot_details to the way they were before the
+ * invocation of createVolumeFromSnapshot(HostVO, SnapshotInfo).
+ */
+ private void deleteVolumeFromSnapshot(SnapshotInfo snapshotInfo) {
+ SnapshotDetailsVO snapshotDetails = handleSnapshotDetails(snapshotInfo.getId(), "tempVolume", "delete");
- result.setResult(errMsg);
+ try {
+ snapshotInfo.getDataStore().getDriver().createAsync(snapshotInfo.getDataStore(), snapshotInfo, null);
+ }
+ finally {
+ _snapshotDetailsDao.remove(snapshotDetails.getId());
+ }
+ }
- callback.complete(result);
+ private SnapshotDetailsVO handleSnapshotDetails(long csSnapshotId, String name, String value) {
+ _snapshotDetailsDao.removeDetail(csSnapshotId, name);
- return null;
+ SnapshotDetailsVO snapshotDetails = new SnapshotDetailsVO(csSnapshotId, name, value, false);
+
+ return _snapshotDetailsDao.persist(snapshotDetails);
}
- private Map<String, String> getSnapshotDetails(StoragePoolVO storagePoolVO, SnapshotInfo snapshotInfo) {
- Map<String, String> details = new HashMap<String, String>();
+ private boolean canStorageSystemCreateVolumeFromVolume(SnapshotInfo snapshotInfo) {
+ boolean supportsCloningVolumeFromVolume = false;
- details.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
- details.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
+ DataStore dataStore = _dataStoreMgr.getDataStore(snapshotInfo.getDataStore().getId(), DataStoreRole.Primary);
- long snapshotId = snapshotInfo.getId();
+ Map<String, String> mapCapabilities = dataStore.getDriver().getCapabilities();
- details.put(DiskTO.IQN, getProperty(snapshotId, DiskTO.IQN));
+ if (mapCapabilities != null) {
+ String value = mapCapabilities.get(DataStoreCapabilities.CAN_CREATE_VOLUME_FROM_VOLUME.toString());
- details.put(DiskTO.CHAP_INITIATOR_USERNAME, getProperty(snapshotId, DiskTO.CHAP_INITIATOR_USERNAME));
- details.put(DiskTO.CHAP_INITIATOR_SECRET, getProperty(snapshotId, DiskTO.CHAP_INITIATOR_SECRET));
- details.put(DiskTO.CHAP_TARGET_USERNAME, getProperty(snapshotId, DiskTO.CHAP_TARGET_USERNAME));
- details.put(DiskTO.CHAP_TARGET_SECRET, getProperty(snapshotId, DiskTO.CHAP_TARGET_SECRET));
+ supportsCloningVolumeFromVolume = new Boolean(value);
--- End diff --
``new Boolean`` avoid the constant pool -- putting unnecessary pressure on the garbage collector. Please use ``Boolean.valueOf`` instead.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r59074068
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -80,32 +92,48 @@
@Inject private AgentManager _agentMgr;
@Inject private ConfigurationDao _configDao;
+ @Inject private DataStoreManager _dataStoreMgr;
@Inject private DiskOfferingDao _diskOfferingDao;
@Inject private HostDao _hostDao;
+ @Inject private HostDetailsDao _hostDetailsDao;
@Inject private ManagementService _mgr;
@Inject private PrimaryDataStoreDao _storagePoolDao;
@Inject private SnapshotDao _snapshotDao;
@Inject private SnapshotDetailsDao _snapshotDetailsDao;
@Inject private VolumeDao _volumeDao;
@Inject private VolumeDataFactory _volumeDataFactory;
+ @Inject private VolumeDetailsDao _volumeDetailsDao;
@Inject private VolumeService _volumeService;
@Override
public StrategyPriority canHandle(DataObject srcData, DataObject destData) {
if (srcData instanceof SnapshotInfo) {
- if (canHandle(srcData.getDataStore()) || canHandle(destData.getDataStore())) {
+ if (canHandle(srcData) || canHandle(destData)) {
return StrategyPriority.HIGHEST;
}
}
+ if (srcData instanceof TemplateInfo && destData instanceof VolumeInfo &&
+ (srcData.getDataStore().getId() == destData.getDataStore().getId()) &&
+ (canHandle(srcData) || canHandle(destData))) {
+ // Both source and dest are on the same storage, so just clone them.
+ return StrategyPriority.HIGHEST;
+ }
+
return StrategyPriority.CANT_HANDLE;
}
- private boolean canHandle(DataStore dataStore) {
+ private boolean canHandle(DataObject dataObject) {
--- End diff --
What is ``dataObject`` is ``null``?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r59074456
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -132,31 +171,52 @@ public void copyAsync(DataObject srcData, DataObject destData, Host destHost, As
validate(snapshotInfo);
- boolean canHandleSrc = canHandle(srcData.getDataStore());
+ boolean canHandleSrc = canHandle(srcData);
if (canHandleSrc && destData instanceof TemplateInfo &&
(destData.getDataStore().getRole() == DataStoreRole.Image || destData.getDataStore().getRole() == DataStoreRole.ImageCache)) {
handleCreateTemplateFromSnapshot(snapshotInfo, (TemplateInfo)destData, callback);
+
return;
}
if (destData instanceof VolumeInfo) {
VolumeInfo volumeInfo = (VolumeInfo)destData;
- boolean canHandleDest = canHandle(destData.getDataStore());
+
+ boolean canHandleDest = canHandle(destData);
if (canHandleSrc && canHandleDest) {
- handleCreateVolumeFromSnapshotBothOnStorageSystem(snapshotInfo, volumeInfo, callback);
- return;
+ if (snapshotInfo.getDataStore().getId() == volumeInfo.getDataStore().getId()) {
+ handleCreateVolumeFromSnapshotBothOnStorageSystem(snapshotInfo, volumeInfo, callback);
+ return;
+ }
+ else {
+ throw new UnsupportedOperationException("This operation is not supported (DataStoreCapabilities.STORAGE_SYSTEM_SNAPSHOT " +
--- End diff --
To aid debugging, please include the name of the destination storage plugin.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by rodrigo93 <gi...@git.apache.org>.
Github user rodrigo93 commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r53033391
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -172,6 +195,12 @@ private void validate(SnapshotInfo snapshotInfo) {
}
}
+ private boolean usingBackendSnapshotFor(SnapshotInfo snapshotInfo) {
+ String property = getProperty(snapshotInfo.getId(), "takeSnapshot");
+
+ return Boolean.parseBoolean(property);
+ }
+
private Void handleCreateTemplateFromSnapshot(SnapshotInfo snapshotInfo, TemplateInfo templateInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
--- End diff --
I see... Using "Void" instead of "void" is not a good programming practice and should be "avoided" =P.
Another suggestion that I forgot to mention is to squash your commits into one.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on the pull request:
https://github.com/apache/cloudstack/pull/1403#issuecomment-207554891
@mike-tutkowski why can't the Marbin test cases be checked in? While we can't run them, it would be nice to review them to understand exactly how the feature is being tested. We can exclude them by default to avoid running into problems on full runs that don't have SolidFire hardware available.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on the pull request:
https://github.com/apache/cloudstack/pull/1403#issuecomment-207554051
@mike-tutkowski @swill I don't think a formal guideline has been set, but we have been pushing toward adopting more idiomatic Java naming conventions. The use of ``_``, ``m_``, and ``s_`` prefixes for class members is not idiomatic. Generally, Java instance members are camel case and static members are all uppercase. It would also be nice to adopt the more stand naming of ``LOGGER`` or ``LOG`` rather than ``s_logger``. If it is not too much trouble, I would encourage you to remove these prefixes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r59076982
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -172,78 +232,211 @@ private void validate(SnapshotInfo snapshotInfo) {
}
}
- private Void handleCreateTemplateFromSnapshot(SnapshotInfo snapshotInfo, TemplateInfo templateInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
+ private boolean usingBackendSnapshotFor(SnapshotInfo snapshotInfo) {
+ String property = getProperty(snapshotInfo.getId(), "takeSnapshot");
+
+ return Boolean.parseBoolean(property);
+ }
+
+ private void handleCreateTemplateFromSnapshot(SnapshotInfo snapshotInfo, TemplateInfo templateInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
try {
snapshotInfo.processEvent(Event.CopyingRequested);
}
catch (Exception ex) {
throw new CloudRuntimeException("This snapshot is not currently in a state where it can be used to create a template.");
}
- HostVO hostVO = getHost(snapshotInfo.getDataStore().getId());
- DataStore srcDataStore = snapshotInfo.getDataStore();
-
- String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
- int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
- CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), templateInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+ HostVO hostVO = getHost(snapshotInfo);
- String errMsg = null;
+ boolean usingBackendSnapshot = usingBackendSnapshotFor(snapshotInfo);
+ boolean computeClusterSupportsResign = computeClusterSupportsResign(hostVO.getClusterId());
- CopyCmdAnswer copyCmdAnswer = null;
+ if (usingBackendSnapshot && !computeClusterSupportsResign) {
+ throw new CloudRuntimeException("Unable to locate an applicable host with which to perform a resignature operation");
+ }
try {
- _volumeService.grantAccess(snapshotInfo, hostVO, srcDataStore);
+ if (usingBackendSnapshot) {
+ createVolumeFromSnapshot(hostVO, snapshotInfo, true);
+ }
- Map<String, String> srcDetails = getSnapshotDetails(_storagePoolDao.findById(srcDataStore.getId()), snapshotInfo);
+ DataStore srcDataStore = snapshotInfo.getDataStore();
- copyCommand.setOptions(srcDetails);
+ String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
+ int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
+ CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), templateInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+
+ String errMsg = null;
+
+ CopyCmdAnswer copyCmdAnswer = null;
- copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
- }
- catch (Exception ex) {
- throw new CloudRuntimeException(ex.getMessage());
- }
- finally {
try {
- _volumeService.revokeAccess(snapshotInfo, hostVO, srcDataStore);
+ // If we are using a back-end snapshot, then we should still have access to it from the hosts in the cluster that hostVO is in
+ // (because we passed in true as the third parameter to createVolumeFromSnapshot above).
+ if (usingBackendSnapshot == false) {
+ _volumeService.grantAccess(snapshotInfo, hostVO, srcDataStore);
+ }
+
+ Map<String, String> srcDetails = getSnapshotDetails(snapshotInfo);
+
+ copyCommand.setOptions(srcDetails);
+
+ copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
}
catch (Exception ex) {
- s_logger.debug(ex.getMessage(), ex);
+ throw new CloudRuntimeException(ex.getMessage());
}
+ finally {
+ try {
+ _volumeService.revokeAccess(snapshotInfo, hostVO, srcDataStore);
+ }
+ catch (Exception ex) {
+ s_logger.debug(ex.getMessage(), ex);
+ }
- if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
- if (copyCmdAnswer != null && copyCmdAnswer.getDetails() != null && !copyCmdAnswer.getDetails().isEmpty()) {
- errMsg = copyCmdAnswer.getDetails();
+ if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
+ if (copyCmdAnswer != null && !StringUtils.isEmpty(copyCmdAnswer.getDetails())) {
+ errMsg = copyCmdAnswer.getDetails();
+ }
+ else {
+ errMsg = "Unable to perform host-side operation";
+ }
}
- else {
- errMsg = "Unable to perform host-side operation";
+
+ try {
+ if (errMsg == null) {
+ snapshotInfo.processEvent(Event.OperationSuccessed);
+ }
+ else {
+ snapshotInfo.processEvent(Event.OperationFailed);
+ }
+ }
+ catch (Exception ex) {
+ s_logger.debug(ex.getMessage(), ex);
}
}
- try {
- if (errMsg == null) {
- snapshotInfo.processEvent(Event.OperationSuccessed);
+ CopyCommandResult result = new CopyCommandResult(null, copyCmdAnswer);
+
+ result.setResult(errMsg);
+
+ callback.complete(result);
+ }
+ finally {
+ if (usingBackendSnapshot) {
+ deleteVolumeFromSnapshot(snapshotInfo);
+ }
+ }
+ }
+
+ /**
+ * Clones a template present on the storage to a new volume and resignatures it.
+ *
+ * @param srcData source template
+ * @param destData destination ROOT volume
+ * @param callback for async
+ */
+ private void handleCreateVolumeFromTemplateBothOnStorageSystem(DataObject srcData, DataObject destData, AsyncCompletionCallback<CopyCommandResult> callback) {
+ CopyCmdAnswer copyCmdAnswer = null;
+ String errMsg = null;
+
+ TemplateInfo templateInfo = (TemplateInfo)srcData;
+ VolumeInfo volumeInfo = (VolumeInfo)destData;
+
+ HostVO hostVO = getHost(volumeInfo.getDataCenterId(), true);
+
+ if (hostVO == null) {
+ throw new CloudRuntimeException("Unable to locate a host capable of resigning in the zone with the following ID: " + volumeInfo.getDataCenterId());
+ }
+
+ boolean computeClusterSupportsResign = computeClusterSupportsResign(hostVO.getClusterId());
+
+ if (!computeClusterSupportsResign) {
+ throw new CloudRuntimeException("Unable to locate an applicable host with which to perform a resignature operation");
+ }
+
+ try {
+ VolumeDetailVO volumeDetail = new VolumeDetailVO(volumeInfo.getId(),
+ "cloneOfTemplate",
+ String.valueOf(templateInfo.getId()),
+ false);
+
+ volumeDetail = _volumeDetailsDao.persist(volumeDetail);
+
+ AsyncCallFuture<VolumeApiResult> future = _volumeService.createVolumeAsync(volumeInfo, volumeInfo.getDataStore());
+ VolumeApiResult result = future.get();
+
+ if (volumeDetail != null) {
+ _volumeDetailsDao.remove(volumeDetail.getId());
+ }
+
+ if (result.isFailed()) {
+ s_logger.debug("Failed to create a volume: " + result.getResult());
+
+ throw new CloudRuntimeException(result.getResult());
+ }
+
+ volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
+
+ volumeInfo.processEvent(Event.MigrationRequested);
+
+ volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
+
+ copyCmdAnswer = performResignature(volumeInfo, hostVO);
+
+ if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
+ if (copyCmdAnswer != null && !StringUtils.isEmpty(copyCmdAnswer.getDetails())) {
+ throw new CloudRuntimeException(copyCmdAnswer.getDetails());
}
else {
- snapshotInfo.processEvent(Event.OperationFailed);
+ throw new CloudRuntimeException("Unable to perform host-side operation");
}
}
- catch (Exception ex) {
- s_logger.debug(ex.getMessage(), ex);
+ } catch (Exception ex) {
+ try {
+ volumeInfo.getDataStore().getDriver().deleteAsync(volumeInfo.getDataStore(), volumeInfo, null);
+ } catch (Exception e) {
+ s_logger.debug("Exception while deleting volume on the backend " + e.getMessage());
--- End diff --
Would information about this error not be useful to an operator? If would be, please consider logging to ``ERROR``. Also, please include more context information about the host, volume, and template involved in the operation to aid operational debugging.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by mike-tutkowski <gi...@git.apache.org>.
Github user mike-tutkowski commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r59074279
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -80,32 +92,48 @@
@Inject private AgentManager _agentMgr;
@Inject private ConfigurationDao _configDao;
+ @Inject private DataStoreManager _dataStoreMgr;
@Inject private DiskOfferingDao _diskOfferingDao;
@Inject private HostDao _hostDao;
+ @Inject private HostDetailsDao _hostDetailsDao;
@Inject private ManagementService _mgr;
@Inject private PrimaryDataStoreDao _storagePoolDao;
@Inject private SnapshotDao _snapshotDao;
@Inject private SnapshotDetailsDao _snapshotDetailsDao;
@Inject private VolumeDao _volumeDao;
@Inject private VolumeDataFactory _volumeDataFactory;
+ @Inject private VolumeDetailsDao _volumeDetailsDao;
@Inject private VolumeService _volumeService;
@Override
public StrategyPriority canHandle(DataObject srcData, DataObject destData) {
if (srcData instanceof SnapshotInfo) {
- if (canHandle(srcData.getDataStore()) || canHandle(destData.getDataStore())) {
+ if (canHandle(srcData) || canHandle(destData)) {
return StrategyPriority.HIGHEST;
}
}
+ if (srcData instanceof TemplateInfo && destData instanceof VolumeInfo &&
+ (srcData.getDataStore().getId() == destData.getDataStore().getId()) &&
+ (canHandle(srcData) || canHandle(destData))) {
+ // Both source and dest are on the same storage, so just clone them.
+ return StrategyPriority.HIGHEST;
+ }
+
return StrategyPriority.CANT_HANDLE;
}
- private boolean canHandle(DataStore dataStore) {
+ private boolean canHandle(DataObject dataObject) {
--- End diff --
I would say that that is an infrastructure fail on the part of CloudStack (passing in null there). We could defend against it, but then we should probably do so everywhere where canHandle is implemented.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r59077320
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -255,99 +448,153 @@ private Void handleCreateVolumeFromSnapshotBothOnStorageSystem(SnapshotInfo snap
VolumeApiResult result = future.get();
+ if (volumeDetail != null) {
+ _volumeDetailsDao.remove(volumeDetail.getId());
+ }
+
if (result.isFailed()) {
s_logger.debug("Failed to create a volume: " + result.getResult());
throw new CloudRuntimeException(result.getResult());
}
- }
- catch (Exception ex) {
- throw new CloudRuntimeException(ex.getMessage());
- }
- volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
+ volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
- volumeInfo.processEvent(Event.MigrationRequested);
+ volumeInfo.processEvent(Event.MigrationRequested);
- volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
+ volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
- HostVO hostVO = getHost(snapshotInfo.getDataStore().getId());
+ if (useCloning) {
+ copyCmdAnswer = performResignature(volumeInfo, hostVO);
+ }
+ else {
+ // asking for a XenServer host here so we don't always prefer to use XenServer hosts that support resigning
+ // even when we don't need those hosts to do this kind of copy work
+ hostVO = getHost(snapshotInfo.getDataCenterId(), false);
- String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
- int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
- CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), volumeInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+ copyCmdAnswer = performCopyOfVdi(volumeInfo, snapshotInfo, hostVO);
+ }
- CopyCmdAnswer copyCmdAnswer = null;
+ if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
+ if (copyCmdAnswer != null && !StringUtils.isEmpty(copyCmdAnswer.getDetails())) {
+ errMsg = copyCmdAnswer.getDetails();
+ }
+ else {
+ errMsg = "Unable to perform host-side operation";
+ }
+ }
+ }
+ catch (Exception ex) {
+ errMsg = ex.getMessage() != null ? ex.getMessage() : "Copy operation failed";
--- End diff --
Please include more detail about the operation being performed to assist operational debugging.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by mike-tutkowski <gi...@git.apache.org>.
Github user mike-tutkowski commented on the pull request:
https://github.com/apache/cloudstack/pull/1403#issuecomment-208120039
@jburwell Unfortunately Google's ImmutableMap doesn't support null for a value (which we use). I noticed this while running regression tests and googled to see if there is a way around it. Google suggested using an unmodifiableMap, but it doesn't provide the same benefits (we wouldn't be making a copying of the underlying data). It does allow you to provide back to callers a read-only implementation, though.
I could allows iterate over the passed-in list and only add to the ImmutableMap entries that don't have null for a value. Not super elegant, but it would work.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r59073181
--- Diff: core/src/org/apache/cloudstack/storage/command/ResignatureCommand.java ---
@@ -0,0 +1,46 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+
+package org.apache.cloudstack.storage.command;
+
+import java.util.Map;
+
+public final class ResignatureCommand extends StorageSubSystemCommand {
+ private final Map<String, String> _details;
+
+ private boolean _executeInSequence = true;
+
+ public ResignatureCommand(final Map<String, String> details) {
+ _details = details;
+ }
+
+ public Map<String, String> getDetails() {
+ return _details;
--- End diff --
Return a copy of ``_details`` map to avoid any side-effects by the caller making subsequence changes to the map.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by mike-tutkowski <gi...@git.apache.org>.
Github user mike-tutkowski commented on the pull request:
https://github.com/apache/cloudstack/pull/1403#issuecomment-217228739
@jburwell Just wanted to point out that an earlier PR of mine, #816, has four Marvin integration tests (three out of four of those apply to this PR, too). I wanted to mention that so you know they are going into the CloudStack repo (even though they require SolidFire hardware to run successfully). There are a few methods in the tests that I'd like to extract to a utils module, but I plan to do in the near future as I've been spending a lot more time building out Marvin tests in general lately. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by mike-tutkowski <gi...@git.apache.org>.
Github user mike-tutkowski commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r63132177
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -172,78 +258,207 @@ private void validate(SnapshotInfo snapshotInfo) {
}
}
- private Void handleCreateTemplateFromSnapshot(SnapshotInfo snapshotInfo, TemplateInfo templateInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
+ private boolean usingBackendSnapshotFor(SnapshotInfo snapshotInfo) {
+ String property = getProperty(snapshotInfo.getId(), "takeSnapshot");
+
+ return Boolean.parseBoolean(property);
+ }
+
+ private void handleCreateTemplateFromSnapshot(SnapshotInfo snapshotInfo, TemplateInfo templateInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
try {
snapshotInfo.processEvent(Event.CopyingRequested);
}
catch (Exception ex) {
throw new CloudRuntimeException("This snapshot is not currently in a state where it can be used to create a template.");
}
- HostVO hostVO = getHost(snapshotInfo.getDataStore().getId());
- DataStore srcDataStore = snapshotInfo.getDataStore();
+ HostVO hostVO = getHost(snapshotInfo);
- String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
- int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
- CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), templateInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+ boolean usingBackendSnapshot = usingBackendSnapshotFor(snapshotInfo);
+ boolean computeClusterSupportsResign = clusterDao.computeWhetherClusterSupportsResigning(hostVO.getClusterId());
- String errMsg = null;
-
- CopyCmdAnswer copyCmdAnswer = null;
+ if (usingBackendSnapshot && !computeClusterSupportsResign) {
+ throw new CloudRuntimeException("Unable to locate an applicable host with which to perform a resignature operation");
+ }
try {
- _volumeService.grantAccess(snapshotInfo, hostVO, srcDataStore);
+ if (usingBackendSnapshot) {
+ createVolumeFromSnapshot(hostVO, snapshotInfo, true);
+ }
- Map<String, String> srcDetails = getSnapshotDetails(_storagePoolDao.findById(srcDataStore.getId()), snapshotInfo);
+ DataStore srcDataStore = snapshotInfo.getDataStore();
- copyCommand.setOptions(srcDetails);
+ String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
+ int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
+ CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), templateInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+
+ String errMsg = null;
+
+ CopyCmdAnswer copyCmdAnswer = null;
- copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
- }
- catch (Exception ex) {
- throw new CloudRuntimeException(ex.getMessage());
- }
- finally {
try {
- _volumeService.revokeAccess(snapshotInfo, hostVO, srcDataStore);
+ // If we are using a back-end snapshot, then we should still have access to it from the hosts in the cluster that hostVO is in
+ // (because we passed in true as the third parameter to createVolumeFromSnapshot above).
+ if (usingBackendSnapshot == false) {
+ _volumeService.grantAccess(snapshotInfo, hostVO, srcDataStore);
+ }
+
+ Map<String, String> srcDetails = getSnapshotDetails(snapshotInfo);
+
+ copyCommand.setOptions(srcDetails);
+
+ copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
}
- catch (Exception ex) {
- s_logger.debug(ex.getMessage(), ex);
+ catch (AgentUnavailableException | OperationTimedoutException ex) {
+ throw new CloudRuntimeException("Failed to create template from snapshot : " + ex.getMessage());
--- End diff --
Done
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by DaanHoogland <gi...@git.apache.org>.
Github user DaanHoogland commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r62868269
--- Diff: engine/schema/src/com/cloud/dc/dao/ClusterDaoImpl.java ---
@@ -260,4 +268,41 @@ public boolean remove(Long id) {
sc.setParameters("dataCenterId", zoneId);
return customSearch(sc, null);
}
+
+ @Override
+ public boolean computeClusterSupportsResign(long clusterId) {
--- End diff --
sugested name change: computeWhetherClusterSupportsResign or even computeWhetherClusterSupportsResigning
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by DaanHoogland <gi...@git.apache.org>.
Github user DaanHoogland commented on the pull request:
https://github.com/apache/cloudstack/pull/1403#issuecomment-218540891
Did a review, I have some remarks, some questions as well.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by DaanHoogland <gi...@git.apache.org>.
Github user DaanHoogland commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r63082184
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -172,78 +232,211 @@ private void validate(SnapshotInfo snapshotInfo) {
}
}
- private Void handleCreateTemplateFromSnapshot(SnapshotInfo snapshotInfo, TemplateInfo templateInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
+ private boolean usingBackendSnapshotFor(SnapshotInfo snapshotInfo) {
+ String property = getProperty(snapshotInfo.getId(), "takeSnapshot");
+
+ return Boolean.parseBoolean(property);
+ }
+
+ private void handleCreateTemplateFromSnapshot(SnapshotInfo snapshotInfo, TemplateInfo templateInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
try {
snapshotInfo.processEvent(Event.CopyingRequested);
}
catch (Exception ex) {
throw new CloudRuntimeException("This snapshot is not currently in a state where it can be used to create a template.");
}
- HostVO hostVO = getHost(snapshotInfo.getDataStore().getId());
- DataStore srcDataStore = snapshotInfo.getDataStore();
-
- String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
- int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
- CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), templateInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+ HostVO hostVO = getHost(snapshotInfo);
- String errMsg = null;
+ boolean usingBackendSnapshot = usingBackendSnapshotFor(snapshotInfo);
+ boolean computeClusterSupportsResign = computeClusterSupportsResign(hostVO.getClusterId());
- CopyCmdAnswer copyCmdAnswer = null;
+ if (usingBackendSnapshot && !computeClusterSupportsResign) {
+ throw new CloudRuntimeException("Unable to locate an applicable host with which to perform a resignature operation");
+ }
try {
- _volumeService.grantAccess(snapshotInfo, hostVO, srcDataStore);
+ if (usingBackendSnapshot) {
+ createVolumeFromSnapshot(hostVO, snapshotInfo, true);
+ }
- Map<String, String> srcDetails = getSnapshotDetails(_storagePoolDao.findById(srcDataStore.getId()), snapshotInfo);
+ DataStore srcDataStore = snapshotInfo.getDataStore();
- copyCommand.setOptions(srcDetails);
+ String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
+ int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
+ CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), templateInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+
+ String errMsg = null;
+
+ CopyCmdAnswer copyCmdAnswer = null;
- copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
- }
- catch (Exception ex) {
- throw new CloudRuntimeException(ex.getMessage());
- }
- finally {
try {
- _volumeService.revokeAccess(snapshotInfo, hostVO, srcDataStore);
+ // If we are using a back-end snapshot, then we should still have access to it from the hosts in the cluster that hostVO is in
+ // (because we passed in true as the third parameter to createVolumeFromSnapshot above).
+ if (usingBackendSnapshot == false) {
+ _volumeService.grantAccess(snapshotInfo, hostVO, srcDataStore);
+ }
+
+ Map<String, String> srcDetails = getSnapshotDetails(snapshotInfo);
+
+ copyCommand.setOptions(srcDetails);
+
+ copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
}
catch (Exception ex) {
- s_logger.debug(ex.getMessage(), ex);
+ throw new CloudRuntimeException(ex.getMessage());
}
+ finally {
+ try {
+ _volumeService.revokeAccess(snapshotInfo, hostVO, srcDataStore);
+ }
+ catch (Exception ex) {
+ s_logger.debug(ex.getMessage(), ex);
+ }
- if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
- if (copyCmdAnswer != null && copyCmdAnswer.getDetails() != null && !copyCmdAnswer.getDetails().isEmpty()) {
- errMsg = copyCmdAnswer.getDetails();
+ if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
+ if (copyCmdAnswer != null && !StringUtils.isEmpty(copyCmdAnswer.getDetails())) {
+ errMsg = copyCmdAnswer.getDetails();
+ }
+ else {
+ errMsg = "Unable to perform host-side operation";
+ }
}
- else {
- errMsg = "Unable to perform host-side operation";
+
+ try {
+ if (errMsg == null) {
+ snapshotInfo.processEvent(Event.OperationSuccessed);
+ }
+ else {
+ snapshotInfo.processEvent(Event.OperationFailed);
+ }
+ }
+ catch (Exception ex) {
+ s_logger.debug(ex.getMessage(), ex);
}
}
- try {
- if (errMsg == null) {
- snapshotInfo.processEvent(Event.OperationSuccessed);
+ CopyCommandResult result = new CopyCommandResult(null, copyCmdAnswer);
+
+ result.setResult(errMsg);
+
+ callback.complete(result);
+ }
+ finally {
+ if (usingBackendSnapshot) {
+ deleteVolumeFromSnapshot(snapshotInfo);
+ }
+ }
+ }
+
+ /**
+ * Clones a template present on the storage to a new volume and resignatures it.
+ *
+ * @param srcData source template
+ * @param destData destination ROOT volume
+ * @param callback for async
+ */
+ private void handleCreateVolumeFromTemplateBothOnStorageSystem(DataObject srcData, DataObject destData, AsyncCompletionCallback<CopyCommandResult> callback) {
+ CopyCmdAnswer copyCmdAnswer = null;
+ String errMsg = null;
+
+ TemplateInfo templateInfo = (TemplateInfo)srcData;
+ VolumeInfo volumeInfo = (VolumeInfo)destData;
--- End diff --
So did you change it? I still see them as DataObject.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by DaanHoogland <gi...@git.apache.org>.
Github user DaanHoogland commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r62881152
--- Diff: engine/storage/volume/src/org/apache/cloudstack/storage/volume/VolumeServiceImpl.java ---
@@ -554,6 +574,51 @@ protected Void managedCopyBaseImageCallback(AsyncCallbackDispatcher<VolumeServic
return null;
}
+ protected Void createManagedTemplateImageCallback(AsyncCallbackDispatcher<VolumeServiceImpl, CreateCmdResult> callback, CreateVolumeContext<CreateCmdResult> context) {
+ CreateCmdResult result = callback.getResult();
+ VolumeApiResult res = new VolumeApiResult(null);
+
+ res.setResult(result.getResult());
+
+ AsyncCallFuture<VolumeApiResult> future = context.getFuture();
+ DataObject templateOnPrimaryStoreObj = context.getVolume();
+
+ if (result.isSuccess()) {
+ ((TemplateObject)templateOnPrimaryStoreObj).setInstallPath(result.getPath());
+ templateOnPrimaryStoreObj.processEvent(Event.OperationSuccessed, result.getAnswer());
+ }
+ else {
+ templateOnPrimaryStoreObj.processEvent(Event.OperationFailed);
+ }
+
+ future.complete(res);
+
+ return null;
--- End diff --
why Void and return null instead of void and return?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by swill <gi...@git.apache.org>.
Github user swill commented on the pull request:
https://github.com/apache/cloudstack/pull/1403#issuecomment-217481732
@jburwell can I get your final review on this? I think we are probably pretty close on this one. Thanks...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by DaanHoogland <gi...@git.apache.org>.
Github user DaanHoogland commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r62875028
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -172,78 +254,214 @@ private void validate(SnapshotInfo snapshotInfo) {
}
}
- private Void handleCreateTemplateFromSnapshot(SnapshotInfo snapshotInfo, TemplateInfo templateInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
+ private boolean usingBackendSnapshotFor(SnapshotInfo snapshotInfo) {
+ String property = getProperty(snapshotInfo.getId(), "takeSnapshot");
+
+ return Boolean.parseBoolean(property);
+ }
+
+ private void handleCreateTemplateFromSnapshot(SnapshotInfo snapshotInfo, TemplateInfo templateInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
try {
snapshotInfo.processEvent(Event.CopyingRequested);
}
catch (Exception ex) {
throw new CloudRuntimeException("This snapshot is not currently in a state where it can be used to create a template.");
}
- HostVO hostVO = getHost(snapshotInfo.getDataStore().getId());
- DataStore srcDataStore = snapshotInfo.getDataStore();
+ HostVO hostVO = getHost(snapshotInfo);
- String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
- int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
- CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), templateInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
-
- String errMsg = null;
+ boolean usingBackendSnapshot = usingBackendSnapshotFor(snapshotInfo);
+ boolean computeClusterSupportsResign = _clusterDao.computeClusterSupportsResign(hostVO.getClusterId());
- CopyCmdAnswer copyCmdAnswer = null;
+ if (usingBackendSnapshot && !computeClusterSupportsResign) {
+ throw new CloudRuntimeException("Unable to locate an applicable host with which to perform a resignature operation");
+ }
try {
- _volumeService.grantAccess(snapshotInfo, hostVO, srcDataStore);
+ if (usingBackendSnapshot) {
+ createVolumeFromSnapshot(hostVO, snapshotInfo, true);
+ }
- Map<String, String> srcDetails = getSnapshotDetails(_storagePoolDao.findById(srcDataStore.getId()), snapshotInfo);
+ DataStore srcDataStore = snapshotInfo.getDataStore();
- copyCommand.setOptions(srcDetails);
+ String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
+ int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
+ CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), templateInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+
+ String errMsg = null;
+
+ CopyCmdAnswer copyCmdAnswer = null;
- copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
- }
- catch (Exception ex) {
- throw new CloudRuntimeException(ex.getMessage());
- }
- finally {
try {
- _volumeService.revokeAccess(snapshotInfo, hostVO, srcDataStore);
+ // If we are using a back-end snapshot, then we should still have access to it from the hosts in the cluster that hostVO is in
+ // (because we passed in true as the third parameter to createVolumeFromSnapshot above).
+ if (usingBackendSnapshot == false) {
+ _volumeService.grantAccess(snapshotInfo, hostVO, srcDataStore);
+ }
+
+ Map<String, String> srcDetails = getSnapshotDetails(snapshotInfo);
+
+ copyCommand.setOptions(srcDetails);
+
+ copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
}
catch (Exception ex) {
- s_logger.debug(ex.getMessage(), ex);
+ throw new CloudRuntimeException("Failed to create template from snapshot : " + ex.getMessage());
}
+ finally {
+ try {
+ _volumeService.revokeAccess(snapshotInfo, hostVO, srcDataStore);
+ }
+ catch (Exception ex) {
+ s_logger.warn(ex.getMessage(), ex);
+ }
- if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
- if (copyCmdAnswer != null && copyCmdAnswer.getDetails() != null && !copyCmdAnswer.getDetails().isEmpty()) {
- errMsg = copyCmdAnswer.getDetails();
+ if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
+ if (copyCmdAnswer != null && !StringUtils.isEmpty(copyCmdAnswer.getDetails())) {
+ errMsg = copyCmdAnswer.getDetails();
+ }
+ else {
+ errMsg = "Unable to create template from snapshot";
+ }
}
- else {
- errMsg = "Unable to perform host-side operation";
+
+ try {
+ if (StringUtils.isEmpty(errMsg)) {
+ snapshotInfo.processEvent(Event.OperationSuccessed);
+ }
+ else {
+ snapshotInfo.processEvent(Event.OperationFailed);
+ }
+ }
+ catch (Exception ex) {
+ s_logger.warn(ex.getMessage(), ex);
}
}
- try {
- if (errMsg == null) {
- snapshotInfo.processEvent(Event.OperationSuccessed);
+ CopyCommandResult result = new CopyCommandResult(null, copyCmdAnswer);
+
+ result.setResult(errMsg);
+
+ callback.complete(result);
+ }
+ finally {
+ if (usingBackendSnapshot) {
+ deleteVolumeFromSnapshot(snapshotInfo);
+ }
+ }
+ }
+
+ /**
+ * Clones a template present on the storage to a new volume and resignatures it.
+ *
+ * @param srcData source template
+ * @param destData destination ROOT volume
+ * @param callback for async
+ */
+ private void handleCreateVolumeFromTemplateBothOnStorageSystem(DataObject srcData, DataObject destData, AsyncCompletionCallback<CopyCommandResult> callback) {
+ Preconditions.checkArgument(srcData != null, "Passing 'null' to srcData of handleCreateVolumeFromTemplateBothOnStorageSystem is not supported.");
+ Preconditions.checkArgument(destData != null, "Passing 'null' to destData of handleCreateVolumeFromTemplateBothOnStorageSystem is not supported.");
+
+ CopyCmdAnswer copyCmdAnswer = null;
+ String errMsg = null;
+
+ TemplateInfo templateInfo = (TemplateInfo)srcData;
+ VolumeInfo volumeInfo = (VolumeInfo)destData;
+
+ HostVO hostVO = getHost(volumeInfo.getDataCenterId(), true);
+
+ if (hostVO == null) {
+ throw new CloudRuntimeException("Unable to locate a host capable of resigning in the zone with the following ID: " + volumeInfo.getDataCenterId());
+ }
+
+ boolean computeClusterSupportsResign = _clusterDao.computeClusterSupportsResign(hostVO.getClusterId());
+
+ if (!computeClusterSupportsResign) {
+ throw new CloudRuntimeException("Unable to locate an applicable host with which to perform a resignature operation");
+ }
+
+ try {
+ VolumeDetailVO volumeDetail = new VolumeDetailVO(volumeInfo.getId(),
+ "cloneOfTemplate",
+ String.valueOf(templateInfo.getId()),
+ false);
+
+ volumeDetail = _volumeDetailsDao.persist(volumeDetail);
+
+ AsyncCallFuture<VolumeApiResult> future = _volumeService.createVolumeAsync(volumeInfo, volumeInfo.getDataStore());
+ VolumeApiResult result = future.get();
+
+ if (volumeDetail != null) {
+ _volumeDetailsDao.remove(volumeDetail.getId());
+ }
+
+ if (result.isFailed()) {
+ s_logger.warn("Failed to create a volume: " + result.getResult());
+
+ throw new CloudRuntimeException(result.getResult());
+ }
+
+ volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
+
+ volumeInfo.processEvent(Event.MigrationRequested);
+
+ volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
+
+ copyCmdAnswer = performResignature(volumeInfo, hostVO);
+
+ if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
+ if (copyCmdAnswer != null && !StringUtils.isEmpty(copyCmdAnswer.getDetails())) {
+ throw new CloudRuntimeException(copyCmdAnswer.getDetails());
}
else {
- snapshotInfo.processEvent(Event.OperationFailed);
+ throw new CloudRuntimeException("Unable to create a volume from a template");
}
}
- catch (Exception ex) {
- s_logger.debug(ex.getMessage(), ex);
+ } catch (Exception ex) {
+ try {
+ volumeInfo.getDataStore().getDriver().deleteAsync(volumeInfo.getDataStore(), volumeInfo, null);
+ } catch (Exception e) {
+ s_logger.warn("Exception while deleting volume on the backend " + e.getMessage());
}
+
+ throw new CloudRuntimeException("Create volume from template (ID = " + templateInfo.getId() + ") failed " + ex.getMessage());
}
CopyCommandResult result = new CopyCommandResult(null, copyCmdAnswer);
result.setResult(errMsg);
callback.complete(result);
-
- return null;
}
- private Void handleCreateVolumeFromSnapshotBothOnStorageSystem(SnapshotInfo snapshotInfo, VolumeInfo volumeInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
+ private void handleCreateVolumeFromSnapshotBothOnStorageSystem(SnapshotInfo snapshotInfo, VolumeInfo volumeInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
--- End diff --
this method has become 75 lines. let's split it (and the likes)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by DaanHoogland <gi...@git.apache.org>.
Github user DaanHoogland commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r62872442
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -172,78 +232,211 @@ private void validate(SnapshotInfo snapshotInfo) {
}
}
- private Void handleCreateTemplateFromSnapshot(SnapshotInfo snapshotInfo, TemplateInfo templateInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
+ private boolean usingBackendSnapshotFor(SnapshotInfo snapshotInfo) {
+ String property = getProperty(snapshotInfo.getId(), "takeSnapshot");
+
+ return Boolean.parseBoolean(property);
+ }
+
+ private void handleCreateTemplateFromSnapshot(SnapshotInfo snapshotInfo, TemplateInfo templateInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
try {
snapshotInfo.processEvent(Event.CopyingRequested);
}
catch (Exception ex) {
throw new CloudRuntimeException("This snapshot is not currently in a state where it can be used to create a template.");
}
- HostVO hostVO = getHost(snapshotInfo.getDataStore().getId());
- DataStore srcDataStore = snapshotInfo.getDataStore();
-
- String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
- int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
- CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), templateInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+ HostVO hostVO = getHost(snapshotInfo);
- String errMsg = null;
+ boolean usingBackendSnapshot = usingBackendSnapshotFor(snapshotInfo);
+ boolean computeClusterSupportsResign = computeClusterSupportsResign(hostVO.getClusterId());
- CopyCmdAnswer copyCmdAnswer = null;
+ if (usingBackendSnapshot && !computeClusterSupportsResign) {
+ throw new CloudRuntimeException("Unable to locate an applicable host with which to perform a resignature operation");
+ }
try {
- _volumeService.grantAccess(snapshotInfo, hostVO, srcDataStore);
+ if (usingBackendSnapshot) {
+ createVolumeFromSnapshot(hostVO, snapshotInfo, true);
+ }
- Map<String, String> srcDetails = getSnapshotDetails(_storagePoolDao.findById(srcDataStore.getId()), snapshotInfo);
+ DataStore srcDataStore = snapshotInfo.getDataStore();
- copyCommand.setOptions(srcDetails);
+ String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
+ int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
+ CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), templateInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+
+ String errMsg = null;
+
+ CopyCmdAnswer copyCmdAnswer = null;
- copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
- }
- catch (Exception ex) {
- throw new CloudRuntimeException(ex.getMessage());
- }
- finally {
try {
--- End diff --
I can not see what Rafael means. I think the code has changed since he made the comment. I do see this method as growing out of proportion so some refactorring for readability and maintainability seems in order.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by DaanHoogland <gi...@git.apache.org>.
Github user DaanHoogland commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r62876283
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -361,59 +603,221 @@ private String getProperty(long snapshotId, String property) {
}
private Map<String, String> getVolumeDetails(VolumeInfo volumeInfo) {
- Map<String, String> sourceDetails = new HashMap<String, String>();
+ Map<String, String> volumeDetails = new HashMap<String, String>();
VolumeVO volumeVO = _volumeDao.findById(volumeInfo.getId());
long storagePoolId = volumeVO.getPoolId();
StoragePoolVO storagePoolVO = _storagePoolDao.findById(storagePoolId);
- sourceDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
- sourceDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
- sourceDetails.put(DiskTO.IQN, volumeVO.get_iScsiName());
+ volumeDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
+ volumeDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
+ volumeDetails.put(DiskTO.IQN, volumeVO.get_iScsiName());
ChapInfo chapInfo = _volumeService.getChapInfo(volumeInfo, volumeInfo.getDataStore());
if (chapInfo != null) {
- sourceDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, chapInfo.getInitiatorUsername());
- sourceDetails.put(DiskTO.CHAP_INITIATOR_SECRET, chapInfo.getInitiatorSecret());
- sourceDetails.put(DiskTO.CHAP_TARGET_USERNAME, chapInfo.getTargetUsername());
- sourceDetails.put(DiskTO.CHAP_TARGET_SECRET, chapInfo.getTargetSecret());
+ volumeDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, chapInfo.getInitiatorUsername());
+ volumeDetails.put(DiskTO.CHAP_INITIATOR_SECRET, chapInfo.getInitiatorSecret());
+ volumeDetails.put(DiskTO.CHAP_TARGET_USERNAME, chapInfo.getTargetUsername());
+ volumeDetails.put(DiskTO.CHAP_TARGET_SECRET, chapInfo.getTargetSecret());
}
- return sourceDetails;
+ return volumeDetails;
+ }
+
+ private Map<String, String> getSnapshotDetails(SnapshotInfo snapshotInfo) {
+ Map<String, String> snapshotDetails = new HashMap<String, String>();
+
+ long storagePoolId = snapshotInfo.getDataStore().getId();
+ StoragePoolVO storagePoolVO = _storagePoolDao.findById(storagePoolId);
+
+ snapshotDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
+ snapshotDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
+
+ long snapshotId = snapshotInfo.getId();
+
+ snapshotDetails.put(DiskTO.IQN, getProperty(snapshotId, DiskTO.IQN));
+
+ snapshotDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, getProperty(snapshotId, DiskTO.CHAP_INITIATOR_USERNAME));
+ snapshotDetails.put(DiskTO.CHAP_INITIATOR_SECRET, getProperty(snapshotId, DiskTO.CHAP_INITIATOR_SECRET));
+ snapshotDetails.put(DiskTO.CHAP_TARGET_USERNAME, getProperty(snapshotId, DiskTO.CHAP_TARGET_USERNAME));
+ snapshotDetails.put(DiskTO.CHAP_TARGET_SECRET, getProperty(snapshotId, DiskTO.CHAP_TARGET_SECRET));
+
+ return snapshotDetails;
}
- public HostVO getHost(long dataStoreId) {
- StoragePoolVO storagePoolVO = _storagePoolDao.findById(dataStoreId);
+ private HostVO getHost(SnapshotInfo snapshotInfo) {
+ HostVO hostVO = getHost(snapshotInfo.getDataCenterId(), true);
- List<? extends Cluster> clusters = _mgr.searchForClusters(storagePoolVO.getDataCenterId(), new Long(0), Long.MAX_VALUE, HypervisorType.XenServer.toString());
+ if (hostVO == null) {
+ hostVO = getHost(snapshotInfo.getDataCenterId(), false);
- if (clusters == null) {
- throw new CloudRuntimeException("Unable to locate an applicable cluster");
+ if (hostVO == null) {
+ throw new CloudRuntimeException("Unable to locate an applicable host in data center with ID = " + snapshotInfo.getDataCenterId());
+ }
}
- for (Cluster cluster : clusters) {
- if (cluster.getAllocationState() == AllocationState.Enabled) {
- List<HostVO> hosts = _hostDao.findByClusterId(cluster.getId());
+ return hostVO;
+ }
- if (hosts != null) {
- for (HostVO host : hosts) {
- if (host.getResourceState() == ResourceState.Enabled) {
- return host;
- }
- }
+ private HostVO getHost(Long zoneId, boolean computeClusterMustSupportResign) {
+ Preconditions.checkArgument(zoneId != null, "Zone ID cannot be null.");
+
+ List<HostVO> hosts = _hostDao.listByDataCenterIdAndHypervisorType(zoneId, HypervisorType.XenServer);
+
+ if (hosts == null) {
+ return null;
+ }
+
+ List<Long> clustersToSkip = new ArrayList<>();
+
+ Collections.shuffle(hosts, random);
+
+ for (HostVO host : hosts) {
+ if (computeClusterMustSupportResign) {
+ long clusterId = host.getClusterId();
+
+ if (clustersToSkip.contains(clusterId)) {
+ continue;
+ }
+
+ if (_clusterDao.computeClusterSupportsResign(clusterId)) {
+ return host;
+ }
+ else {
+ clustersToSkip.add(clusterId);
}
}
+ else {
+ return host;
+ }
}
- throw new CloudRuntimeException("Unable to locate an applicable cluster");
+ return null;
}
@Override
public void copyAsync(Map<VolumeInfo, DataStore> volumeMap, VirtualMachineTO vmTo, Host srcHost, Host destHost, AsyncCompletionCallback<CopyCommandResult> callback) {
CopyCommandResult result = new CopyCommandResult(null, null);
+
result.setResult("Unsupported operation requested for copying data.");
+
callback.complete(result);
}
+
+ private Map<String, String> getDetails(DataObject dataObj) {
+ if (dataObj instanceof VolumeInfo) {
+ return getVolumeDetails((VolumeInfo)dataObj);
+ }
+ else if (dataObj instanceof SnapshotInfo) {
+ return getSnapshotDetails((SnapshotInfo)dataObj);
+ }
+
+ throw new CloudRuntimeException("'dataObj' must be of type 'VolumeInfo' or 'SnapshotInfo'.");
+ }
+
+ private CopyCmdAnswer performResignature(DataObject dataObj, HostVO hostVO) {
+ return performResignature(dataObj, hostVO, false);
+ }
+
+ private CopyCmdAnswer performResignature(DataObject dataObj, HostVO hostVO, boolean keepGrantedAccess) {
--- End diff --
same name here, @mike-tutkowski ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by DaanHoogland <gi...@git.apache.org>.
Github user DaanHoogland commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r62891804
--- Diff: server/src/com/cloud/storage/StorageManagerImpl.java ---
@@ -209,6 +211,8 @@
@Inject
protected HostDao _hostDao;
@Inject
+ private HostDetailsDao _hostDetailsDao;
--- End diff --
unused
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r63110833
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -55,65 +68,98 @@
import com.cloud.host.Host;
import com.cloud.host.HostVO;
import com.cloud.host.dao.HostDao;
+import com.cloud.host.dao.HostDetailsDao;
import com.cloud.hypervisor.Hypervisor.HypervisorType;
-import com.cloud.org.Cluster;
-import com.cloud.org.Grouping.AllocationState;
-import com.cloud.resource.ResourceState;
import com.cloud.server.ManagementService;
import com.cloud.storage.DataStoreRole;
import com.cloud.storage.DiskOfferingVO;
import com.cloud.storage.SnapshotVO;
import com.cloud.storage.Storage.ImageFormat;
+import com.cloud.storage.VolumeDetailVO;
import com.cloud.storage.VolumeVO;
import com.cloud.storage.dao.DiskOfferingDao;
import com.cloud.storage.dao.SnapshotDao;
import com.cloud.storage.dao.SnapshotDetailsDao;
import com.cloud.storage.dao.SnapshotDetailsVO;
import com.cloud.storage.dao.VolumeDao;
+import com.cloud.storage.dao.VolumeDetailsDao;
import com.cloud.utils.NumbersUtil;
import com.cloud.utils.exception.CloudRuntimeException;
import com.cloud.vm.VirtualMachineManager;
+import com.google.common.base.Preconditions;
+
@Component
public class StorageSystemDataMotionStrategy implements DataMotionStrategy {
private static final Logger s_logger = Logger.getLogger(StorageSystemDataMotionStrategy.class);
+ private static final Random random = new Random(System.nanoTime());
@Inject private AgentManager _agentMgr;
@Inject private ConfigurationDao _configDao;
+ @Inject private DataStoreManager dataStoreMgr;
@Inject private DiskOfferingDao _diskOfferingDao;
+ @Inject private ClusterDao clusterDao;
@Inject private HostDao _hostDao;
+ @Inject private HostDetailsDao hostDetailsDao;
@Inject private ManagementService _mgr;
@Inject private PrimaryDataStoreDao _storagePoolDao;
@Inject private SnapshotDao _snapshotDao;
@Inject private SnapshotDetailsDao _snapshotDetailsDao;
@Inject private VolumeDao _volumeDao;
@Inject private VolumeDataFactory _volumeDataFactory;
+ @Inject private VolumeDetailsDao volumeDetailsDao;
@Inject private VolumeService _volumeService;
@Override
public StrategyPriority canHandle(DataObject srcData, DataObject destData) {
if (srcData instanceof SnapshotInfo) {
- if (canHandle(srcData.getDataStore()) || canHandle(destData.getDataStore())) {
+ if (canHandle(srcData) || canHandle(destData)) {
return StrategyPriority.HIGHEST;
}
}
+ if (srcData instanceof TemplateInfo && destData instanceof VolumeInfo &&
+ (srcData.getDataStore().getId() == destData.getDataStore().getId()) &&
+ (canHandle(srcData) || canHandle(destData))) {
+ // Both source and dest are on the same storage, so just clone them.
+ return StrategyPriority.HIGHEST;
+ }
+
return StrategyPriority.CANT_HANDLE;
}
- private boolean canHandle(DataStore dataStore) {
+ private boolean canHandle(DataObject dataObject) {
+ Preconditions.checkArgument(dataObject != null, "Passing 'null' to dataObject of canHandle(DataObject) is not supported.");
+
+ DataStore dataStore = dataObject.getDataStore();
+
if (dataStore.getRole() == DataStoreRole.Primary) {
Map<String, String> mapCapabilities = dataStore.getDriver().getCapabilities();
- if (mapCapabilities != null) {
+ if (mapCapabilities == null) {
+ return false;
+ }
+
+ if (dataObject instanceof VolumeInfo || dataObject instanceof SnapshotInfo) {
String value = mapCapabilities.get(DataStoreCapabilities.STORAGE_SYSTEM_SNAPSHOT.toString());
- Boolean supportsStorageSystemSnapshots = new Boolean(value);
+ Boolean supportsStorageSystemSnapshots = Boolean.valueOf(value);
if (supportsStorageSystemSnapshots) {
s_logger.info("Using 'StorageSystemDataMotionStrategy'");
return true;
}
+ } else if (dataObject instanceof TemplateInfo) {
+ // If the storage system can clone volumes, we can cache templates on it.
+ String value = mapCapabilities.get(DataStoreCapabilities.CAN_CREATE_VOLUME_FROM_VOLUME.toString());
+ Boolean canCloneVolume = Boolean.valueOf(value);
+
+ if (canCloneVolume) {
+ s_logger.info("Using 'StorageSystemDataMotionStrategy'");
--- End diff --
Please add contextual information from the ``dataObject`` to help operational debugging.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by mike-tutkowski <gi...@git.apache.org>.
Github user mike-tutkowski commented on the pull request:
https://github.com/apache/cloudstack/pull/1403#issuecomment-210072697
This might be a better link (as it includes the SHA):
https://github.com/apache/cloudstack/tree/ee5370536ac3f87457d5f74adbbb8c6af88da449/plugins/api
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r59077257
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -255,99 +448,153 @@ private Void handleCreateVolumeFromSnapshotBothOnStorageSystem(SnapshotInfo snap
VolumeApiResult result = future.get();
+ if (volumeDetail != null) {
+ _volumeDetailsDao.remove(volumeDetail.getId());
+ }
+
if (result.isFailed()) {
s_logger.debug("Failed to create a volume: " + result.getResult());
throw new CloudRuntimeException(result.getResult());
}
- }
- catch (Exception ex) {
- throw new CloudRuntimeException(ex.getMessage());
- }
- volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
+ volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
- volumeInfo.processEvent(Event.MigrationRequested);
+ volumeInfo.processEvent(Event.MigrationRequested);
- volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
+ volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
- HostVO hostVO = getHost(snapshotInfo.getDataStore().getId());
+ if (useCloning) {
+ copyCmdAnswer = performResignature(volumeInfo, hostVO);
+ }
+ else {
+ // asking for a XenServer host here so we don't always prefer to use XenServer hosts that support resigning
+ // even when we don't need those hosts to do this kind of copy work
+ hostVO = getHost(snapshotInfo.getDataCenterId(), false);
- String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
- int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
- CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), volumeInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+ copyCmdAnswer = performCopyOfVdi(volumeInfo, snapshotInfo, hostVO);
+ }
- CopyCmdAnswer copyCmdAnswer = null;
+ if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
+ if (copyCmdAnswer != null && !StringUtils.isEmpty(copyCmdAnswer.getDetails())) {
+ errMsg = copyCmdAnswer.getDetails();
+ }
+ else {
+ errMsg = "Unable to perform host-side operation";
--- End diff --
Please include more detail about the operation being performed to assist operational debugging.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r59073124
--- Diff: core/src/org/apache/cloudstack/storage/command/ResignatureCommand.java ---
@@ -0,0 +1,46 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+
+package org.apache.cloudstack.storage.command;
+
+import java.util.Map;
+
+public final class ResignatureCommand extends StorageSubSystemCommand {
+ private final Map<String, String> _details;
+
+ private boolean _executeInSequence = true;
+
+ public ResignatureCommand(final Map<String, String> details) {
+ _details = details;
--- End diff --
Copy the ``_details`` map to avoid any side-effects by the caller making subsequence changes to the map.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r59076160
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -172,78 +232,211 @@ private void validate(SnapshotInfo snapshotInfo) {
}
}
- private Void handleCreateTemplateFromSnapshot(SnapshotInfo snapshotInfo, TemplateInfo templateInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
+ private boolean usingBackendSnapshotFor(SnapshotInfo snapshotInfo) {
+ String property = getProperty(snapshotInfo.getId(), "takeSnapshot");
+
+ return Boolean.parseBoolean(property);
+ }
+
+ private void handleCreateTemplateFromSnapshot(SnapshotInfo snapshotInfo, TemplateInfo templateInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
try {
snapshotInfo.processEvent(Event.CopyingRequested);
}
catch (Exception ex) {
throw new CloudRuntimeException("This snapshot is not currently in a state where it can be used to create a template.");
}
- HostVO hostVO = getHost(snapshotInfo.getDataStore().getId());
- DataStore srcDataStore = snapshotInfo.getDataStore();
-
- String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
- int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
- CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), templateInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+ HostVO hostVO = getHost(snapshotInfo);
- String errMsg = null;
+ boolean usingBackendSnapshot = usingBackendSnapshotFor(snapshotInfo);
+ boolean computeClusterSupportsResign = computeClusterSupportsResign(hostVO.getClusterId());
- CopyCmdAnswer copyCmdAnswer = null;
+ if (usingBackendSnapshot && !computeClusterSupportsResign) {
+ throw new CloudRuntimeException("Unable to locate an applicable host with which to perform a resignature operation");
+ }
try {
- _volumeService.grantAccess(snapshotInfo, hostVO, srcDataStore);
+ if (usingBackendSnapshot) {
+ createVolumeFromSnapshot(hostVO, snapshotInfo, true);
+ }
- Map<String, String> srcDetails = getSnapshotDetails(_storagePoolDao.findById(srcDataStore.getId()), snapshotInfo);
+ DataStore srcDataStore = snapshotInfo.getDataStore();
- copyCommand.setOptions(srcDetails);
+ String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
+ int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
+ CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), templateInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+
+ String errMsg = null;
+
+ CopyCmdAnswer copyCmdAnswer = null;
- copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
- }
- catch (Exception ex) {
- throw new CloudRuntimeException(ex.getMessage());
- }
- finally {
try {
- _volumeService.revokeAccess(snapshotInfo, hostVO, srcDataStore);
+ // If we are using a back-end snapshot, then we should still have access to it from the hosts in the cluster that hostVO is in
+ // (because we passed in true as the third parameter to createVolumeFromSnapshot above).
+ if (usingBackendSnapshot == false) {
+ _volumeService.grantAccess(snapshotInfo, hostVO, srcDataStore);
+ }
+
+ Map<String, String> srcDetails = getSnapshotDetails(snapshotInfo);
+
+ copyCommand.setOptions(srcDetails);
+
+ copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
}
catch (Exception ex) {
- s_logger.debug(ex.getMessage(), ex);
+ throw new CloudRuntimeException(ex.getMessage());
}
+ finally {
+ try {
+ _volumeService.revokeAccess(snapshotInfo, hostVO, srcDataStore);
+ }
+ catch (Exception ex) {
+ s_logger.debug(ex.getMessage(), ex);
+ }
- if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
- if (copyCmdAnswer != null && copyCmdAnswer.getDetails() != null && !copyCmdAnswer.getDetails().isEmpty()) {
- errMsg = copyCmdAnswer.getDetails();
+ if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
+ if (copyCmdAnswer != null && !StringUtils.isEmpty(copyCmdAnswer.getDetails())) {
+ errMsg = copyCmdAnswer.getDetails();
+ }
+ else {
+ errMsg = "Unable to perform host-side operation";
+ }
}
- else {
- errMsg = "Unable to perform host-side operation";
+
+ try {
+ if (errMsg == null) {
+ snapshotInfo.processEvent(Event.OperationSuccessed);
+ }
+ else {
+ snapshotInfo.processEvent(Event.OperationFailed);
+ }
+ }
+ catch (Exception ex) {
+ s_logger.debug(ex.getMessage(), ex);
}
}
- try {
- if (errMsg == null) {
- snapshotInfo.processEvent(Event.OperationSuccessed);
+ CopyCommandResult result = new CopyCommandResult(null, copyCmdAnswer);
+
+ result.setResult(errMsg);
+
+ callback.complete(result);
+ }
+ finally {
+ if (usingBackendSnapshot) {
+ deleteVolumeFromSnapshot(snapshotInfo);
+ }
+ }
+ }
+
+ /**
+ * Clones a template present on the storage to a new volume and resignatures it.
+ *
+ * @param srcData source template
+ * @param destData destination ROOT volume
+ * @param callback for async
+ */
+ private void handleCreateVolumeFromTemplateBothOnStorageSystem(DataObject srcData, DataObject destData, AsyncCompletionCallback<CopyCommandResult> callback) {
+ CopyCmdAnswer copyCmdAnswer = null;
+ String errMsg = null;
+
+ TemplateInfo templateInfo = (TemplateInfo)srcData;
+ VolumeInfo volumeInfo = (VolumeInfo)destData;
+
+ HostVO hostVO = getHost(volumeInfo.getDataCenterId(), true);
+
+ if (hostVO == null) {
+ throw new CloudRuntimeException("Unable to locate a host capable of resigning in the zone with the following ID: " + volumeInfo.getDataCenterId());
+ }
+
+ boolean computeClusterSupportsResign = computeClusterSupportsResign(hostVO.getClusterId());
+
+ if (!computeClusterSupportsResign) {
+ throw new CloudRuntimeException("Unable to locate an applicable host with which to perform a resignature operation");
+ }
+
+ try {
+ VolumeDetailVO volumeDetail = new VolumeDetailVO(volumeInfo.getId(),
+ "cloneOfTemplate",
+ String.valueOf(templateInfo.getId()),
+ false);
+
+ volumeDetail = _volumeDetailsDao.persist(volumeDetail);
+
+ AsyncCallFuture<VolumeApiResult> future = _volumeService.createVolumeAsync(volumeInfo, volumeInfo.getDataStore());
+ VolumeApiResult result = future.get();
+
+ if (volumeDetail != null) {
+ _volumeDetailsDao.remove(volumeDetail.getId());
+ }
+
+ if (result.isFailed()) {
+ s_logger.debug("Failed to create a volume: " + result.getResult());
+
+ throw new CloudRuntimeException(result.getResult());
+ }
+
+ volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
+
+ volumeInfo.processEvent(Event.MigrationRequested);
+
+ volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
+
+ copyCmdAnswer = performResignature(volumeInfo, hostVO);
+
+ if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
+ if (copyCmdAnswer != null && !StringUtils.isEmpty(copyCmdAnswer.getDetails())) {
+ throw new CloudRuntimeException(copyCmdAnswer.getDetails());
}
else {
- snapshotInfo.processEvent(Event.OperationFailed);
+ throw new CloudRuntimeException("Unable to perform host-side operation");
}
}
- catch (Exception ex) {
- s_logger.debug(ex.getMessage(), ex);
+ } catch (Exception ex) {
--- End diff --
What are all checked and unchecked exceptions being caught?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by mike-tutkowski <gi...@git.apache.org>.
Github user mike-tutkowski commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r52114579
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -255,99 +332,149 @@ private Void handleCreateVolumeFromSnapshotBothOnStorageSystem(SnapshotInfo snap
VolumeApiResult result = future.get();
+ if (volumeDetail != null) {
+ _volumeDetailsDao.remove(volumeDetail.getId());
+ }
+
if (result.isFailed()) {
s_logger.debug("Failed to create a volume: " + result.getResult());
throw new CloudRuntimeException(result.getResult());
}
- }
- catch (Exception ex) {
- throw new CloudRuntimeException(ex.getMessage());
- }
-
- volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
- volumeInfo.processEvent(Event.MigrationRequested);
+ volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
- volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
+ volumeInfo.processEvent(Event.MigrationRequested);
- HostVO hostVO = getHost(snapshotInfo.getDataStore().getId());
+ volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
- String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
- int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
- CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), volumeInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+ if (useCloning) {
+ copyCmdAnswer = performResignature(volumeInfo, hostVO);
+ }
+ else {
+ // asking for a XenServer host here so we don't always prefer to use XenServer hosts that support resigning
+ // even when we don't need those hosts to do this kind of copy work
+ hostVO = getXenServerHost(snapshotInfo.getDataCenterId(), false);
- CopyCmdAnswer copyCmdAnswer = null;
+ copyCmdAnswer = performCopyOfVdi(volumeInfo, snapshotInfo, hostVO);
+ }
- try {
- _volumeService.grantAccess(snapshotInfo, hostVO, snapshotInfo.getDataStore());
- _volumeService.grantAccess(volumeInfo, hostVO, volumeInfo.getDataStore());
+ if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
+ if (copyCmdAnswer != null && copyCmdAnswer.getDetails() != null && !copyCmdAnswer.getDetails().isEmpty()) {
+ errMsg = copyCmdAnswer.getDetails();
+ }
+ else {
+ errMsg = "Unable to perform host-side operation";
+ }
+ }
+ }
+ catch (Exception ex) {
+ errMsg = ex.getMessage() != null ? ex.getMessage() : "Copy operation failed";
+ }
- Map<String, String> srcDetails = getSnapshotDetails(_storagePoolDao.findById(snapshotInfo.getDataStore().getId()), snapshotInfo);
+ CopyCommandResult result = new CopyCommandResult(null, copyCmdAnswer);
- copyCommand.setOptions(srcDetails);
+ result.setResult(errMsg);
- Map<String, String> destDetails = getVolumeDetails(volumeInfo);
+ callback.complete(result);
+ }
- copyCommand.setOptions2(destDetails);
+ // If the underlying storage system is making use of read-only snapshots, this gives the storage system the opportunity to
+ // create a volume from the snapshot so that we can copy the VHD file that should be inside of the snapshot to secondary storage.
+ //
+ // The resultant volume must be writable because we need to resign the SR and the VDI that should be inside of it before we copy
+ // the VHD file to secondary storage.
+ //
+ // If the storage system is using writable snapshots, then nothing need be done by that storage system here because we can just
+ // resign the SR and the VDI that should be inside of the snapshot before copying the VHD file to secondary storage.
+ private void createVolumeFromSnapshot(HostVO hostVO, SnapshotInfo snapshotInfo, boolean keepGrantedAccess) {
--- End diff --
Done - thanks
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r59075010
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -172,78 +232,211 @@ private void validate(SnapshotInfo snapshotInfo) {
}
}
- private Void handleCreateTemplateFromSnapshot(SnapshotInfo snapshotInfo, TemplateInfo templateInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
+ private boolean usingBackendSnapshotFor(SnapshotInfo snapshotInfo) {
+ String property = getProperty(snapshotInfo.getId(), "takeSnapshot");
+
+ return Boolean.parseBoolean(property);
+ }
+
+ private void handleCreateTemplateFromSnapshot(SnapshotInfo snapshotInfo, TemplateInfo templateInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
try {
snapshotInfo.processEvent(Event.CopyingRequested);
}
catch (Exception ex) {
throw new CloudRuntimeException("This snapshot is not currently in a state where it can be used to create a template.");
}
- HostVO hostVO = getHost(snapshotInfo.getDataStore().getId());
- DataStore srcDataStore = snapshotInfo.getDataStore();
-
- String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
- int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
- CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), templateInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+ HostVO hostVO = getHost(snapshotInfo);
- String errMsg = null;
+ boolean usingBackendSnapshot = usingBackendSnapshotFor(snapshotInfo);
+ boolean computeClusterSupportsResign = computeClusterSupportsResign(hostVO.getClusterId());
- CopyCmdAnswer copyCmdAnswer = null;
+ if (usingBackendSnapshot && !computeClusterSupportsResign) {
+ throw new CloudRuntimeException("Unable to locate an applicable host with which to perform a resignature operation");
+ }
try {
- _volumeService.grantAccess(snapshotInfo, hostVO, srcDataStore);
+ if (usingBackendSnapshot) {
+ createVolumeFromSnapshot(hostVO, snapshotInfo, true);
+ }
- Map<String, String> srcDetails = getSnapshotDetails(_storagePoolDao.findById(srcDataStore.getId()), snapshotInfo);
+ DataStore srcDataStore = snapshotInfo.getDataStore();
- copyCommand.setOptions(srcDetails);
+ String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
+ int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
+ CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), templateInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+
+ String errMsg = null;
+
+ CopyCmdAnswer copyCmdAnswer = null;
- copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
- }
- catch (Exception ex) {
- throw new CloudRuntimeException(ex.getMessage());
- }
- finally {
try {
- _volumeService.revokeAccess(snapshotInfo, hostVO, srcDataStore);
+ // If we are using a back-end snapshot, then we should still have access to it from the hosts in the cluster that hostVO is in
+ // (because we passed in true as the third parameter to createVolumeFromSnapshot above).
+ if (usingBackendSnapshot == false) {
+ _volumeService.grantAccess(snapshotInfo, hostVO, srcDataStore);
+ }
+
+ Map<String, String> srcDetails = getSnapshotDetails(snapshotInfo);
+
+ copyCommand.setOptions(srcDetails);
+
+ copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
}
catch (Exception ex) {
- s_logger.debug(ex.getMessage(), ex);
+ throw new CloudRuntimeException(ex.getMessage());
}
+ finally {
+ try {
+ _volumeService.revokeAccess(snapshotInfo, hostVO, srcDataStore);
+ }
+ catch (Exception ex) {
+ s_logger.debug(ex.getMessage(), ex);
--- End diff --
Why not log the ignored exception to ``WARN``?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by mike-tutkowski <gi...@git.apache.org>.
Github user mike-tutkowski commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r63083184
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -361,59 +603,221 @@ private String getProperty(long snapshotId, String property) {
}
private Map<String, String> getVolumeDetails(VolumeInfo volumeInfo) {
- Map<String, String> sourceDetails = new HashMap<String, String>();
+ Map<String, String> volumeDetails = new HashMap<String, String>();
VolumeVO volumeVO = _volumeDao.findById(volumeInfo.getId());
long storagePoolId = volumeVO.getPoolId();
StoragePoolVO storagePoolVO = _storagePoolDao.findById(storagePoolId);
- sourceDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
- sourceDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
- sourceDetails.put(DiskTO.IQN, volumeVO.get_iScsiName());
+ volumeDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
+ volumeDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
+ volumeDetails.put(DiskTO.IQN, volumeVO.get_iScsiName());
ChapInfo chapInfo = _volumeService.getChapInfo(volumeInfo, volumeInfo.getDataStore());
if (chapInfo != null) {
- sourceDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, chapInfo.getInitiatorUsername());
- sourceDetails.put(DiskTO.CHAP_INITIATOR_SECRET, chapInfo.getInitiatorSecret());
- sourceDetails.put(DiskTO.CHAP_TARGET_USERNAME, chapInfo.getTargetUsername());
- sourceDetails.put(DiskTO.CHAP_TARGET_SECRET, chapInfo.getTargetSecret());
+ volumeDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, chapInfo.getInitiatorUsername());
+ volumeDetails.put(DiskTO.CHAP_INITIATOR_SECRET, chapInfo.getInitiatorSecret());
+ volumeDetails.put(DiskTO.CHAP_TARGET_USERNAME, chapInfo.getTargetUsername());
+ volumeDetails.put(DiskTO.CHAP_TARGET_SECRET, chapInfo.getTargetSecret());
}
- return sourceDetails;
+ return volumeDetails;
+ }
+
+ private Map<String, String> getSnapshotDetails(SnapshotInfo snapshotInfo) {
+ Map<String, String> snapshotDetails = new HashMap<String, String>();
+
+ long storagePoolId = snapshotInfo.getDataStore().getId();
+ StoragePoolVO storagePoolVO = _storagePoolDao.findById(storagePoolId);
+
+ snapshotDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
+ snapshotDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
+
+ long snapshotId = snapshotInfo.getId();
+
+ snapshotDetails.put(DiskTO.IQN, getProperty(snapshotId, DiskTO.IQN));
+
+ snapshotDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, getProperty(snapshotId, DiskTO.CHAP_INITIATOR_USERNAME));
+ snapshotDetails.put(DiskTO.CHAP_INITIATOR_SECRET, getProperty(snapshotId, DiskTO.CHAP_INITIATOR_SECRET));
+ snapshotDetails.put(DiskTO.CHAP_TARGET_USERNAME, getProperty(snapshotId, DiskTO.CHAP_TARGET_USERNAME));
+ snapshotDetails.put(DiskTO.CHAP_TARGET_SECRET, getProperty(snapshotId, DiskTO.CHAP_TARGET_SECRET));
+
+ return snapshotDetails;
}
- public HostVO getHost(long dataStoreId) {
- StoragePoolVO storagePoolVO = _storagePoolDao.findById(dataStoreId);
+ private HostVO getHost(SnapshotInfo snapshotInfo) {
+ HostVO hostVO = getHost(snapshotInfo.getDataCenterId(), true);
- List<? extends Cluster> clusters = _mgr.searchForClusters(storagePoolVO.getDataCenterId(), new Long(0), Long.MAX_VALUE, HypervisorType.XenServer.toString());
+ if (hostVO == null) {
+ hostVO = getHost(snapshotInfo.getDataCenterId(), false);
- if (clusters == null) {
- throw new CloudRuntimeException("Unable to locate an applicable cluster");
+ if (hostVO == null) {
+ throw new CloudRuntimeException("Unable to locate an applicable host in data center with ID = " + snapshotInfo.getDataCenterId());
+ }
}
- for (Cluster cluster : clusters) {
- if (cluster.getAllocationState() == AllocationState.Enabled) {
- List<HostVO> hosts = _hostDao.findByClusterId(cluster.getId());
+ return hostVO;
+ }
- if (hosts != null) {
- for (HostVO host : hosts) {
- if (host.getResourceState() == ResourceState.Enabled) {
- return host;
- }
- }
+ private HostVO getHost(Long zoneId, boolean computeClusterMustSupportResign) {
+ Preconditions.checkArgument(zoneId != null, "Zone ID cannot be null.");
+
+ List<HostVO> hosts = _hostDao.listByDataCenterIdAndHypervisorType(zoneId, HypervisorType.XenServer);
+
+ if (hosts == null) {
+ return null;
+ }
+
+ List<Long> clustersToSkip = new ArrayList<>();
+
+ Collections.shuffle(hosts, random);
+
+ for (HostVO host : hosts) {
+ if (computeClusterMustSupportResign) {
+ long clusterId = host.getClusterId();
+
+ if (clustersToSkip.contains(clusterId)) {
+ continue;
+ }
+
+ if (_clusterDao.computeClusterSupportsResign(clusterId)) {
+ return host;
+ }
+ else {
+ clustersToSkip.add(clusterId);
}
}
+ else {
+ return host;
+ }
}
- throw new CloudRuntimeException("Unable to locate an applicable cluster");
+ return null;
}
@Override
public void copyAsync(Map<VolumeInfo, DataStore> volumeMap, VirtualMachineTO vmTo, Host srcHost, Host destHost, AsyncCompletionCallback<CopyCommandResult> callback) {
CopyCommandResult result = new CopyCommandResult(null, null);
+
result.setResult("Unsupported operation requested for copying data.");
+
callback.complete(result);
}
+
+ private Map<String, String> getDetails(DataObject dataObj) {
+ if (dataObj instanceof VolumeInfo) {
+ return getVolumeDetails((VolumeInfo)dataObj);
+ }
+ else if (dataObj instanceof SnapshotInfo) {
+ return getSnapshotDetails((SnapshotInfo)dataObj);
+ }
+
+ throw new CloudRuntimeException("'dataObj' must be of type 'VolumeInfo' or 'SnapshotInfo'.");
+ }
+
+ private CopyCmdAnswer performResignature(DataObject dataObj, HostVO hostVO) {
--- End diff --
@DaanHoogland It's kind of like "Perform Re-signature Operation".
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by mike-tutkowski <gi...@git.apache.org>.
Github user mike-tutkowski commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r63132007
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -172,78 +258,207 @@ private void validate(SnapshotInfo snapshotInfo) {
}
}
- private Void handleCreateTemplateFromSnapshot(SnapshotInfo snapshotInfo, TemplateInfo templateInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
+ private boolean usingBackendSnapshotFor(SnapshotInfo snapshotInfo) {
+ String property = getProperty(snapshotInfo.getId(), "takeSnapshot");
+
+ return Boolean.parseBoolean(property);
+ }
+
+ private void handleCreateTemplateFromSnapshot(SnapshotInfo snapshotInfo, TemplateInfo templateInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
try {
snapshotInfo.processEvent(Event.CopyingRequested);
}
catch (Exception ex) {
throw new CloudRuntimeException("This snapshot is not currently in a state where it can be used to create a template.");
}
- HostVO hostVO = getHost(snapshotInfo.getDataStore().getId());
- DataStore srcDataStore = snapshotInfo.getDataStore();
+ HostVO hostVO = getHost(snapshotInfo);
- String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
- int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
- CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), templateInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+ boolean usingBackendSnapshot = usingBackendSnapshotFor(snapshotInfo);
+ boolean computeClusterSupportsResign = clusterDao.computeWhetherClusterSupportsResigning(hostVO.getClusterId());
- String errMsg = null;
-
- CopyCmdAnswer copyCmdAnswer = null;
+ if (usingBackendSnapshot && !computeClusterSupportsResign) {
+ throw new CloudRuntimeException("Unable to locate an applicable host with which to perform a resignature operation");
--- End diff --
Done
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r59077350
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -255,99 +448,153 @@ private Void handleCreateVolumeFromSnapshotBothOnStorageSystem(SnapshotInfo snap
VolumeApiResult result = future.get();
+ if (volumeDetail != null) {
+ _volumeDetailsDao.remove(volumeDetail.getId());
+ }
+
if (result.isFailed()) {
s_logger.debug("Failed to create a volume: " + result.getResult());
throw new CloudRuntimeException(result.getResult());
}
- }
- catch (Exception ex) {
- throw new CloudRuntimeException(ex.getMessage());
- }
- volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
+ volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
- volumeInfo.processEvent(Event.MigrationRequested);
+ volumeInfo.processEvent(Event.MigrationRequested);
- volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
+ volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
- HostVO hostVO = getHost(snapshotInfo.getDataStore().getId());
+ if (useCloning) {
+ copyCmdAnswer = performResignature(volumeInfo, hostVO);
+ }
+ else {
+ // asking for a XenServer host here so we don't always prefer to use XenServer hosts that support resigning
+ // even when we don't need those hosts to do this kind of copy work
+ hostVO = getHost(snapshotInfo.getDataCenterId(), false);
- String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
- int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
- CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), volumeInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+ copyCmdAnswer = performCopyOfVdi(volumeInfo, snapshotInfo, hostVO);
+ }
- CopyCmdAnswer copyCmdAnswer = null;
+ if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
+ if (copyCmdAnswer != null && !StringUtils.isEmpty(copyCmdAnswer.getDetails())) {
+ errMsg = copyCmdAnswer.getDetails();
+ }
+ else {
+ errMsg = "Unable to perform host-side operation";
+ }
+ }
+ }
+ catch (Exception ex) {
+ errMsg = ex.getMessage() != null ? ex.getMessage() : "Copy operation failed";
+ }
- try {
- _volumeService.grantAccess(snapshotInfo, hostVO, snapshotInfo.getDataStore());
- _volumeService.grantAccess(volumeInfo, hostVO, volumeInfo.getDataStore());
+ CopyCommandResult result = new CopyCommandResult(null, copyCmdAnswer);
- Map<String, String> srcDetails = getSnapshotDetails(_storagePoolDao.findById(snapshotInfo.getDataStore().getId()), snapshotInfo);
+ result.setResult(errMsg);
- copyCommand.setOptions(srcDetails);
+ callback.complete(result);
+ }
- Map<String, String> destDetails = getVolumeDetails(volumeInfo);
+ /**
+ * If the underlying storage system is making use of read-only snapshots, this gives the storage system the opportunity to
+ * create a volume from the snapshot so that we can copy the VHD file that should be inside of the snapshot to secondary storage.
+ *
+ * The resultant volume must be writable because we need to resign the SR and the VDI that should be inside of it before we copy
+ * the VHD file to secondary storage.
+ *
+ * If the storage system is using writable snapshots, then nothing need be done by that storage system here because we can just
+ * resign the SR and the VDI that should be inside of the snapshot before copying the VHD file to secondary storage.
+ */
+ private void createVolumeFromSnapshot(HostVO hostVO, SnapshotInfo snapshotInfo, boolean keepGrantedAccess) {
+ SnapshotDetailsVO snapshotDetails = handleSnapshotDetails(snapshotInfo.getId(), "tempVolume", "create");
- copyCommand.setOptions2(destDetails);
-
- copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
- }
- catch (Exception ex) {
- throw new CloudRuntimeException(ex.getMessage());
+ try {
+ snapshotInfo.getDataStore().getDriver().createAsync(snapshotInfo.getDataStore(), snapshotInfo, null);
}
finally {
- try {
- _volumeService.revokeAccess(snapshotInfo, hostVO, snapshotInfo.getDataStore());
- }
- catch (Exception ex) {
- s_logger.debug(ex.getMessage(), ex);
- }
-
- try {
- _volumeService.revokeAccess(volumeInfo, hostVO, volumeInfo.getDataStore());
- }
- catch (Exception ex) {
- s_logger.debug(ex.getMessage(), ex);
- }
+ _snapshotDetailsDao.remove(snapshotDetails.getId());
}
- String errMsg = null;
+ CopyCmdAnswer copyCmdAnswer = performResignature(snapshotInfo, hostVO, keepGrantedAccess);
if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
- if (copyCmdAnswer != null && copyCmdAnswer.getDetails() != null && !copyCmdAnswer.getDetails().isEmpty()) {
- errMsg = copyCmdAnswer.getDetails();
+ if (copyCmdAnswer != null && !StringUtils.isEmpty(copyCmdAnswer.getDetails())) {
+ throw new CloudRuntimeException(copyCmdAnswer.getDetails());
}
else {
- errMsg = "Unable to perform host-side operation";
+ throw new CloudRuntimeException("Unable to perform host-side operation");
--- End diff --
Please include more detail about the operation being performed to assist operational debugging.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r59082814
--- Diff: engine/storage/image/src/org/apache/cloudstack/storage/image/store/TemplateObject.java ---
@@ -299,28 +311,17 @@ public DataTO getTO() {
@Override
public String getInstallPath() {
- if (installPath != null)
+ if (installPath != null) {
--- End diff --
Consider using ``Strings.isNullorEmpty`` to cover both the ``null`` and empty cases.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by mike-tutkowski <gi...@git.apache.org>.
Github user mike-tutkowski commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r63080850
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -172,78 +232,211 @@ private void validate(SnapshotInfo snapshotInfo) {
}
}
- private Void handleCreateTemplateFromSnapshot(SnapshotInfo snapshotInfo, TemplateInfo templateInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
+ private boolean usingBackendSnapshotFor(SnapshotInfo snapshotInfo) {
+ String property = getProperty(snapshotInfo.getId(), "takeSnapshot");
+
+ return Boolean.parseBoolean(property);
+ }
+
+ private void handleCreateTemplateFromSnapshot(SnapshotInfo snapshotInfo, TemplateInfo templateInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
try {
snapshotInfo.processEvent(Event.CopyingRequested);
}
catch (Exception ex) {
throw new CloudRuntimeException("This snapshot is not currently in a state where it can be used to create a template.");
}
- HostVO hostVO = getHost(snapshotInfo.getDataStore().getId());
- DataStore srcDataStore = snapshotInfo.getDataStore();
-
- String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
- int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
- CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), templateInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+ HostVO hostVO = getHost(snapshotInfo);
- String errMsg = null;
+ boolean usingBackendSnapshot = usingBackendSnapshotFor(snapshotInfo);
+ boolean computeClusterSupportsResign = computeClusterSupportsResign(hostVO.getClusterId());
- CopyCmdAnswer copyCmdAnswer = null;
+ if (usingBackendSnapshot && !computeClusterSupportsResign) {
+ throw new CloudRuntimeException("Unable to locate an applicable host with which to perform a resignature operation");
+ }
try {
- _volumeService.grantAccess(snapshotInfo, hostVO, srcDataStore);
+ if (usingBackendSnapshot) {
+ createVolumeFromSnapshot(hostVO, snapshotInfo, true);
+ }
- Map<String, String> srcDetails = getSnapshotDetails(_storagePoolDao.findById(srcDataStore.getId()), snapshotInfo);
+ DataStore srcDataStore = snapshotInfo.getDataStore();
- copyCommand.setOptions(srcDetails);
+ String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
+ int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
+ CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), templateInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+
+ String errMsg = null;
+
+ CopyCmdAnswer copyCmdAnswer = null;
- copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
- }
- catch (Exception ex) {
- throw new CloudRuntimeException(ex.getMessage());
- }
- finally {
try {
- _volumeService.revokeAccess(snapshotInfo, hostVO, srcDataStore);
+ // If we are using a back-end snapshot, then we should still have access to it from the hosts in the cluster that hostVO is in
+ // (because we passed in true as the third parameter to createVolumeFromSnapshot above).
+ if (usingBackendSnapshot == false) {
+ _volumeService.grantAccess(snapshotInfo, hostVO, srcDataStore);
+ }
+
+ Map<String, String> srcDetails = getSnapshotDetails(snapshotInfo);
+
+ copyCommand.setOptions(srcDetails);
+
+ copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
}
catch (Exception ex) {
- s_logger.debug(ex.getMessage(), ex);
+ throw new CloudRuntimeException(ex.getMessage());
}
+ finally {
+ try {
+ _volumeService.revokeAccess(snapshotInfo, hostVO, srcDataStore);
+ }
+ catch (Exception ex) {
+ s_logger.debug(ex.getMessage(), ex);
+ }
- if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
- if (copyCmdAnswer != null && copyCmdAnswer.getDetails() != null && !copyCmdAnswer.getDetails().isEmpty()) {
- errMsg = copyCmdAnswer.getDetails();
+ if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
+ if (copyCmdAnswer != null && !StringUtils.isEmpty(copyCmdAnswer.getDetails())) {
+ errMsg = copyCmdAnswer.getDetails();
+ }
+ else {
+ errMsg = "Unable to perform host-side operation";
+ }
}
- else {
- errMsg = "Unable to perform host-side operation";
+
+ try {
+ if (errMsg == null) {
+ snapshotInfo.processEvent(Event.OperationSuccessed);
+ }
+ else {
+ snapshotInfo.processEvent(Event.OperationFailed);
+ }
+ }
+ catch (Exception ex) {
+ s_logger.debug(ex.getMessage(), ex);
}
}
- try {
- if (errMsg == null) {
- snapshotInfo.processEvent(Event.OperationSuccessed);
+ CopyCommandResult result = new CopyCommandResult(null, copyCmdAnswer);
+
+ result.setResult(errMsg);
+
+ callback.complete(result);
+ }
+ finally {
+ if (usingBackendSnapshot) {
+ deleteVolumeFromSnapshot(snapshotInfo);
+ }
+ }
+ }
+
+ /**
+ * Clones a template present on the storage to a new volume and resignatures it.
+ *
+ * @param srcData source template
+ * @param destData destination ROOT volume
+ * @param callback for async
+ */
+ private void handleCreateVolumeFromTemplateBothOnStorageSystem(DataObject srcData, DataObject destData, AsyncCompletionCallback<CopyCommandResult> callback) {
+ CopyCmdAnswer copyCmdAnswer = null;
+ String errMsg = null;
+
+ TemplateInfo templateInfo = (TemplateInfo)srcData;
+ VolumeInfo volumeInfo = (VolumeInfo)destData;
--- End diff --
@jburwell @DaanHoogland Looking at this code again, I see it was for legacy reasons that the first two parameters were of type DataObject.
I was able to switch this to the following signature, which eliminates the need for checking the types passed in:
private void handleCreateVolumeFromTemplateBothOnStorageSystem(TemplateInfo templateInfo, VolumeInfo volumeInfo, AsyncCompletionCallback<CopyCommandResult> callback)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r59077297
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -255,99 +448,153 @@ private Void handleCreateVolumeFromSnapshotBothOnStorageSystem(SnapshotInfo snap
VolumeApiResult result = future.get();
+ if (volumeDetail != null) {
+ _volumeDetailsDao.remove(volumeDetail.getId());
+ }
+
if (result.isFailed()) {
s_logger.debug("Failed to create a volume: " + result.getResult());
throw new CloudRuntimeException(result.getResult());
}
- }
- catch (Exception ex) {
- throw new CloudRuntimeException(ex.getMessage());
- }
- volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
+ volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
- volumeInfo.processEvent(Event.MigrationRequested);
+ volumeInfo.processEvent(Event.MigrationRequested);
- volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
+ volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
- HostVO hostVO = getHost(snapshotInfo.getDataStore().getId());
+ if (useCloning) {
+ copyCmdAnswer = performResignature(volumeInfo, hostVO);
+ }
+ else {
+ // asking for a XenServer host here so we don't always prefer to use XenServer hosts that support resigning
+ // even when we don't need those hosts to do this kind of copy work
+ hostVO = getHost(snapshotInfo.getDataCenterId(), false);
- String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
- int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
- CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), volumeInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+ copyCmdAnswer = performCopyOfVdi(volumeInfo, snapshotInfo, hostVO);
+ }
- CopyCmdAnswer copyCmdAnswer = null;
+ if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
+ if (copyCmdAnswer != null && !StringUtils.isEmpty(copyCmdAnswer.getDetails())) {
+ errMsg = copyCmdAnswer.getDetails();
+ }
+ else {
+ errMsg = "Unable to perform host-side operation";
+ }
+ }
+ }
+ catch (Exception ex) {
--- End diff --
Why are all checked and unchecked exceptions being caught?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by DaanHoogland <gi...@git.apache.org>.
Github user DaanHoogland commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r62890825
--- Diff: plugins/storage/volume/solidfire/src/org/apache/cloudstack/storage/datastore/driver/SolidFirePrimaryDataStoreDriver.java ---
@@ -692,12 +1145,38 @@ private String deleteSnapshot(SnapshotInfo snapshotInfo, long storagePoolId) {
_storagePoolDao.update(storagePoolId, storagePool);
}
catch (Exception ex) {
- s_logger.debug(SolidFireUtil.LOG_PREFIX + "Failed to delete SolidFire volume. CloudStack snapshot ID: " + snapshotId, ex);
+ s_logger.debug(SolidFireUtil.LOG_PREFIX + "Issue in 'deleteSnapshot(SnapshotInfo, long)'. CloudStack snapshot ID: " + csSnapshotId, ex);
- errMsg = ex.getMessage();
+ throw ex;
}
+ }
+
+ private void deleteTemplate(TemplateInfo template, long storagePoolId) {
+ try {
+ SolidFireUtil.SolidFireConnection sfConnection = SolidFireUtil.getSolidFireConnection(storagePoolId, _storagePoolDetailsDao);
+
+ long sfTemplateVolumeId = getVolumeIdFrom_iScsiPath(template.getInstallPath());
- return errMsg;
+ SolidFireUtil.deleteSolidFireVolume(sfConnection, sfTemplateVolumeId);
+
+ VMTemplateStoragePoolVO templatePoolRef = _tmpltPoolDao.findByPoolTemplate(storagePoolId, template.getId());
+
+ _tmpltPoolDao.remove(templatePoolRef.getId());
+
+ StoragePoolVO storagePool = _storagePoolDao.findById(storagePoolId);
+
+ // getUsedBytes(StoragePool) will not include the template to delete because the "template_spool_ref" table has already been updated by this point
+ long usedBytes = getUsedBytes(storagePool);
+
+ storagePool.setUsedBytes(usedBytes < 0 ? 0 : usedBytes);
+
+ _storagePoolDao.update(storagePoolId, storagePool);
+ }
+ catch (Exception ex) {
--- End diff --
???
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by DaanHoogland <gi...@git.apache.org>.
Github user DaanHoogland commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r62876040
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -361,59 +608,225 @@ private String getProperty(long snapshotId, String property) {
}
private Map<String, String> getVolumeDetails(VolumeInfo volumeInfo) {
- Map<String, String> sourceDetails = new HashMap<String, String>();
+ Map<String, String> volumeDetails = new HashMap<String, String>();
VolumeVO volumeVO = _volumeDao.findById(volumeInfo.getId());
long storagePoolId = volumeVO.getPoolId();
StoragePoolVO storagePoolVO = _storagePoolDao.findById(storagePoolId);
- sourceDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
- sourceDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
- sourceDetails.put(DiskTO.IQN, volumeVO.get_iScsiName());
+ volumeDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
+ volumeDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
+ volumeDetails.put(DiskTO.IQN, volumeVO.get_iScsiName());
ChapInfo chapInfo = _volumeService.getChapInfo(volumeInfo, volumeInfo.getDataStore());
if (chapInfo != null) {
- sourceDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, chapInfo.getInitiatorUsername());
- sourceDetails.put(DiskTO.CHAP_INITIATOR_SECRET, chapInfo.getInitiatorSecret());
- sourceDetails.put(DiskTO.CHAP_TARGET_USERNAME, chapInfo.getTargetUsername());
- sourceDetails.put(DiskTO.CHAP_TARGET_SECRET, chapInfo.getTargetSecret());
+ volumeDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, chapInfo.getInitiatorUsername());
+ volumeDetails.put(DiskTO.CHAP_INITIATOR_SECRET, chapInfo.getInitiatorSecret());
+ volumeDetails.put(DiskTO.CHAP_TARGET_USERNAME, chapInfo.getTargetUsername());
+ volumeDetails.put(DiskTO.CHAP_TARGET_SECRET, chapInfo.getTargetSecret());
+ }
+
+ return volumeDetails;
+ }
+
+ private Map<String, String> getSnapshotDetails(SnapshotInfo snapshotInfo) {
+ Map<String, String> snapshotDetails = new HashMap<String, String>();
+
+ long storagePoolId = snapshotInfo.getDataStore().getId();
+ StoragePoolVO storagePoolVO = _storagePoolDao.findById(storagePoolId);
+
+ snapshotDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
+ snapshotDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
+
+ long snapshotId = snapshotInfo.getId();
+
+ snapshotDetails.put(DiskTO.IQN, getProperty(snapshotId, DiskTO.IQN));
+
+ snapshotDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, getProperty(snapshotId, DiskTO.CHAP_INITIATOR_USERNAME));
+ snapshotDetails.put(DiskTO.CHAP_INITIATOR_SECRET, getProperty(snapshotId, DiskTO.CHAP_INITIATOR_SECRET));
+ snapshotDetails.put(DiskTO.CHAP_TARGET_USERNAME, getProperty(snapshotId, DiskTO.CHAP_TARGET_USERNAME));
+ snapshotDetails.put(DiskTO.CHAP_TARGET_SECRET, getProperty(snapshotId, DiskTO.CHAP_TARGET_SECRET));
+
+ return snapshotDetails;
+ }
+
+ private HostVO getHost(SnapshotInfo snapshotInfo) {
+ HostVO hostVO = getHost(snapshotInfo.getDataCenterId(), true);
+
+ if (hostVO == null) {
+ hostVO = getHost(snapshotInfo.getDataCenterId(), false);
+
+ if (hostVO == null) {
+ throw new CloudRuntimeException("Unable to locate an applicable host");
+ }
}
- return sourceDetails;
+ return hostVO;
}
- public HostVO getHost(long dataStoreId) {
- StoragePoolVO storagePoolVO = _storagePoolDao.findById(dataStoreId);
+ private HostVO getHost(Long zoneId, boolean computeClusterMustSupportResign) {
+ if (zoneId == null) {
+ throw new CloudRuntimeException("Zone ID cannot be null.");
+ }
- List<? extends Cluster> clusters = _mgr.searchForClusters(storagePoolVO.getDataCenterId(), new Long(0), Long.MAX_VALUE, HypervisorType.XenServer.toString());
+ List<? extends Cluster> clusters = _mgr.searchForClusters(zoneId, new Long(0), Long.MAX_VALUE, HypervisorType.XenServer.toString());
if (clusters == null) {
- throw new CloudRuntimeException("Unable to locate an applicable cluster");
+ clusters = new ArrayList<>();
}
+ Collections.shuffle(clusters, new Random(System.nanoTime()));
+
+ clusters:
for (Cluster cluster : clusters) {
if (cluster.getAllocationState() == AllocationState.Enabled) {
List<HostVO> hosts = _hostDao.findByClusterId(cluster.getId());
if (hosts != null) {
+ Collections.shuffle(hosts, new Random(System.nanoTime()));
+
for (HostVO host : hosts) {
if (host.getResourceState() == ResourceState.Enabled) {
- return host;
+ if (computeClusterMustSupportResign) {
+ if (computeClusterSupportsResign(cluster.getId())) {
+ return host;
+ }
+ else {
+ // no other host in the cluster in question should be able to satisfy our requirements here, so move on to the next cluster
+ continue clusters;
+ }
+ }
+ else {
+ return host;
+ }
}
}
}
}
}
- throw new CloudRuntimeException("Unable to locate an applicable cluster");
+ return null;
}
@Override
public void copyAsync(Map<VolumeInfo, DataStore> volumeMap, VirtualMachineTO vmTo, Host srcHost, Host destHost, AsyncCompletionCallback<CopyCommandResult> callback) {
CopyCommandResult result = new CopyCommandResult(null, null);
+
result.setResult("Unsupported operation requested for copying data.");
+
callback.complete(result);
}
+
+ private Map<String, String> getDetails(DataObject dataObj) {
+ if (dataObj instanceof VolumeInfo) {
+ return getVolumeDetails((VolumeInfo)dataObj);
+ }
+ else if (dataObj instanceof SnapshotInfo) {
+ return getSnapshotDetails((SnapshotInfo)dataObj);
+ }
--- End diff --
I see your point @jburwell . this is true for canHandle and a couple others as well. Any proposal?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by bvbharatk <gi...@git.apache.org>.
Github user bvbharatk commented on the pull request:
https://github.com/apache/cloudstack/pull/1403#issuecomment-196078641
### ACS CI BVT Run
**Sumarry:**
Build Number 91
Hypervisor xenserver
NetworkType Advanced
Passed=104
Failed=14
Skipped=4
**The follwing tests have known issues**
integration.smoke.test_iso.TestISO.test_07_list_default_iso
integration.smoke.test_privategw_acl.py
integration.smoke.test_vm_snapshots.TestSnapshots.test_01_test_vm_volume_snapshot
integration.smoke.test_vpc_vpn.TestVpcSite2SiteVpn.test_vpc_site2site_vpn
integration.smoke.test_iso.TestISO.test_04_extract_Iso
_Link to logs Folder (search by build_no):_ https://www.dropbox.com/sh/yj3wnzbceo9uef2/AAB6u-Iap-xztdm6jHX9SjPja?dl=0
**Failed tests:**
* integration.smoke.test_vpc_vpn.TestVpcRemoteAccessVpn
* test_vpc_remote_access_vpn
* test_vpc_site2site_vpn
* integration.smoke.test_vm_snapshots.TestSnapshots
* test_01_test_vm_volume_snapshot
* integration.smoke.test_privategw_acl.TestPrivateGwACL
* test_02_vpc_privategw_static_routes
* test_03_rvpc_privategw_static_routes
* integration.smoke.test_guest_vlan_range.TestDedicateGuestVlanRange
* test_dedicateGuestVlanRange
* <nose.suite
* ContextSuite context=TestNiciraContoller>:setup
* integration.smoke.test_primary_storage.TestPrimaryStorageServices
* test_01_primary_storage_iscsi
* <nose.suite
* ContextSuite context=TestCreateVolume>:setup
* integration.smoke.test_internal_lb.TestInternalLb
* test02_internallb_haproxy_stats_on_all_interfaces
* test_01_internallb_roundrobin_1VPC_3VM_HTTP_port80
* integration.smoke.test_templates.TestCreateTemplate
* test_04_extract_template
* integration.smoke.test_iso.TestCreateIso
* test_04_extract_Iso
* test_07_list_default_iso
**Skipped tests:**
test_vm_nic_adapter_vmxnet3
test_deploy_vgpu_enabled_vm
test_06_copy_template
test_06_copy_iso
**Passed test suits:**
integration.smoke.test_deploy_vm_with_userdata.TestDeployVmWithUserData
integration.smoke.test_affinity_groups_projects.TestDeployVmWithAffinityGroup
integration.smoke.test_portable_publicip.TestPortablePublicIPAcquire
integration.smoke.test_over_provisioning.TestUpdateOverProvision
integration.smoke.test_global_settings.TestUpdateConfigWithScope
integration.smoke.test_scale_vm.TestScaleVm
integration.smoke.test_service_offerings.TestCreateServiceOffering
integration.smoke.test_loadbalance.TestLoadBalance
integration.smoke.test_routers.TestRouterServices
integration.smoke.test_reset_vm_on_reboot.TestResetVmOnReboot
integration.smoke.test_snapshots.TestSnapshotRootDisk
integration.smoke.test_deploy_vms_with_varied_deploymentplanners.TestDeployVmWithVariedPlanners
integration.smoke.test_network.TestDeleteAccount
integration.smoke.test_non_contigiousvlan.TestUpdatePhysicalNetwork
integration.smoke.test_deploy_vm_iso.TestDeployVMFromISO
integration.smoke.test_public_ip_range.TestDedicatePublicIPRange
integration.smoke.test_multipleips_per_nic.TestDeployVM
integration.smoke.test_regions.TestRegions
integration.smoke.test_affinity_groups.TestDeployVmWithAffinityGroup
integration.smoke.test_network_acl.TestNetworkACL
integration.smoke.test_pvlan.TestPVLAN
integration.smoke.test_ssvm.TestSSVMs
integration.smoke.test_nic.TestNic
integration.smoke.test_deploy_vm_root_resize.TestDeployVM
integration.smoke.test_resource_detail.TestResourceDetail
integration.smoke.test_secondary_storage.TestSecStorageServices
integration.smoke.test_vm_life_cycle.TestDeployVM
integration.smoke.test_disk_offerings.TestCreateDiskOffering
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r59080710
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -361,59 +608,225 @@ private String getProperty(long snapshotId, String property) {
}
private Map<String, String> getVolumeDetails(VolumeInfo volumeInfo) {
- Map<String, String> sourceDetails = new HashMap<String, String>();
+ Map<String, String> volumeDetails = new HashMap<String, String>();
VolumeVO volumeVO = _volumeDao.findById(volumeInfo.getId());
long storagePoolId = volumeVO.getPoolId();
StoragePoolVO storagePoolVO = _storagePoolDao.findById(storagePoolId);
- sourceDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
- sourceDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
- sourceDetails.put(DiskTO.IQN, volumeVO.get_iScsiName());
+ volumeDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
+ volumeDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
+ volumeDetails.put(DiskTO.IQN, volumeVO.get_iScsiName());
ChapInfo chapInfo = _volumeService.getChapInfo(volumeInfo, volumeInfo.getDataStore());
if (chapInfo != null) {
- sourceDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, chapInfo.getInitiatorUsername());
- sourceDetails.put(DiskTO.CHAP_INITIATOR_SECRET, chapInfo.getInitiatorSecret());
- sourceDetails.put(DiskTO.CHAP_TARGET_USERNAME, chapInfo.getTargetUsername());
- sourceDetails.put(DiskTO.CHAP_TARGET_SECRET, chapInfo.getTargetSecret());
+ volumeDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, chapInfo.getInitiatorUsername());
+ volumeDetails.put(DiskTO.CHAP_INITIATOR_SECRET, chapInfo.getInitiatorSecret());
+ volumeDetails.put(DiskTO.CHAP_TARGET_USERNAME, chapInfo.getTargetUsername());
+ volumeDetails.put(DiskTO.CHAP_TARGET_SECRET, chapInfo.getTargetSecret());
+ }
+
+ return volumeDetails;
+ }
+
+ private Map<String, String> getSnapshotDetails(SnapshotInfo snapshotInfo) {
+ Map<String, String> snapshotDetails = new HashMap<String, String>();
+
+ long storagePoolId = snapshotInfo.getDataStore().getId();
+ StoragePoolVO storagePoolVO = _storagePoolDao.findById(storagePoolId);
+
+ snapshotDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
+ snapshotDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
+
+ long snapshotId = snapshotInfo.getId();
+
+ snapshotDetails.put(DiskTO.IQN, getProperty(snapshotId, DiskTO.IQN));
+
+ snapshotDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, getProperty(snapshotId, DiskTO.CHAP_INITIATOR_USERNAME));
+ snapshotDetails.put(DiskTO.CHAP_INITIATOR_SECRET, getProperty(snapshotId, DiskTO.CHAP_INITIATOR_SECRET));
+ snapshotDetails.put(DiskTO.CHAP_TARGET_USERNAME, getProperty(snapshotId, DiskTO.CHAP_TARGET_USERNAME));
+ snapshotDetails.put(DiskTO.CHAP_TARGET_SECRET, getProperty(snapshotId, DiskTO.CHAP_TARGET_SECRET));
+
+ return snapshotDetails;
+ }
+
+ private HostVO getHost(SnapshotInfo snapshotInfo) {
+ HostVO hostVO = getHost(snapshotInfo.getDataCenterId(), true);
+
+ if (hostVO == null) {
+ hostVO = getHost(snapshotInfo.getDataCenterId(), false);
+
+ if (hostVO == null) {
+ throw new CloudRuntimeException("Unable to locate an applicable host");
+ }
}
- return sourceDetails;
+ return hostVO;
}
- public HostVO getHost(long dataStoreId) {
- StoragePoolVO storagePoolVO = _storagePoolDao.findById(dataStoreId);
+ private HostVO getHost(Long zoneId, boolean computeClusterMustSupportResign) {
+ if (zoneId == null) {
+ throw new CloudRuntimeException("Zone ID cannot be null.");
+ }
- List<? extends Cluster> clusters = _mgr.searchForClusters(storagePoolVO.getDataCenterId(), new Long(0), Long.MAX_VALUE, HypervisorType.XenServer.toString());
+ List<? extends Cluster> clusters = _mgr.searchForClusters(zoneId, new Long(0), Long.MAX_VALUE, HypervisorType.XenServer.toString());
if (clusters == null) {
- throw new CloudRuntimeException("Unable to locate an applicable cluster");
+ clusters = new ArrayList<>();
}
+ Collections.shuffle(clusters, new Random(System.nanoTime()));
+
+ clusters:
for (Cluster cluster : clusters) {
if (cluster.getAllocationState() == AllocationState.Enabled) {
List<HostVO> hosts = _hostDao.findByClusterId(cluster.getId());
if (hosts != null) {
+ Collections.shuffle(hosts, new Random(System.nanoTime()));
+
for (HostVO host : hosts) {
if (host.getResourceState() == ResourceState.Enabled) {
- return host;
+ if (computeClusterMustSupportResign) {
+ if (computeClusterSupportsResign(cluster.getId())) {
+ return host;
+ }
+ else {
+ // no other host in the cluster in question should be able to satisfy our requirements here, so move on to the next cluster
+ continue clusters;
+ }
+ }
+ else {
+ return host;
+ }
}
}
}
}
}
--- End diff --
*I realize that this thought is beyond the scope of this PR, and completely understand if the answer is no.*
Would it be possible to replace the outer ``for`` loop and ``if`` block with a new method on the HostDAO to get all allocated, enabled hosts for a list of clusters? It would not only reduce the load on the database and make the performance profile of this method more deterministic, but it would make it much easier to read.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by mike-tutkowski <gi...@git.apache.org>.
Github user mike-tutkowski commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r52953884
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -172,6 +195,12 @@ private void validate(SnapshotInfo snapshotInfo) {
}
}
+ private boolean usingBackendSnapshotFor(SnapshotInfo snapshotInfo) {
+ String property = getProperty(snapshotInfo.getId(), "takeSnapshot");
+
+ return Boolean.parseBoolean(property);
+ }
+
private Void handleCreateTemplateFromSnapshot(SnapshotInfo snapshotInfo, TemplateInfo templateInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
--- End diff --
Thanks. Initially this method was patterned off of an existing set of methods that used the "Void" approach. I wasn't sure if there was a good reason for why they did it that way and erred on the side of caution and just repeated the pattern.
As it turned out, someone later changed those methods from "Void" to "void" and so I ended up doing so in my sandbox (but haven't yet pushed the changes).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by GabrielBrascher <gi...@git.apache.org>.
Github user GabrielBrascher commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r52108776
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -255,99 +332,149 @@ private Void handleCreateVolumeFromSnapshotBothOnStorageSystem(SnapshotInfo snap
VolumeApiResult result = future.get();
+ if (volumeDetail != null) {
+ _volumeDetailsDao.remove(volumeDetail.getId());
+ }
+
if (result.isFailed()) {
s_logger.debug("Failed to create a volume: " + result.getResult());
throw new CloudRuntimeException(result.getResult());
}
- }
- catch (Exception ex) {
- throw new CloudRuntimeException(ex.getMessage());
- }
-
- volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
- volumeInfo.processEvent(Event.MigrationRequested);
+ volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
- volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
+ volumeInfo.processEvent(Event.MigrationRequested);
- HostVO hostVO = getHost(snapshotInfo.getDataStore().getId());
+ volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
- String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
- int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
- CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), volumeInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+ if (useCloning) {
+ copyCmdAnswer = performResignature(volumeInfo, hostVO);
+ }
+ else {
+ // asking for a XenServer host here so we don't always prefer to use XenServer hosts that support resigning
+ // even when we don't need those hosts to do this kind of copy work
+ hostVO = getXenServerHost(snapshotInfo.getDataCenterId(), false);
- CopyCmdAnswer copyCmdAnswer = null;
+ copyCmdAnswer = performCopyOfVdi(volumeInfo, snapshotInfo, hostVO);
+ }
- try {
- _volumeService.grantAccess(snapshotInfo, hostVO, snapshotInfo.getDataStore());
- _volumeService.grantAccess(volumeInfo, hostVO, volumeInfo.getDataStore());
+ if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
+ if (copyCmdAnswer != null && copyCmdAnswer.getDetails() != null && !copyCmdAnswer.getDetails().isEmpty()) {
+ errMsg = copyCmdAnswer.getDetails();
+ }
+ else {
+ errMsg = "Unable to perform host-side operation";
+ }
+ }
+ }
+ catch (Exception ex) {
+ errMsg = ex.getMessage() != null ? ex.getMessage() : "Copy operation failed";
+ }
- Map<String, String> srcDetails = getSnapshotDetails(_storagePoolDao.findById(snapshotInfo.getDataStore().getId()), snapshotInfo);
+ CopyCommandResult result = new CopyCommandResult(null, copyCmdAnswer);
- copyCommand.setOptions(srcDetails);
+ result.setResult(errMsg);
- Map<String, String> destDetails = getVolumeDetails(volumeInfo);
+ callback.complete(result);
+ }
- copyCommand.setOptions2(destDetails);
+ // If the underlying storage system is making use of read-only snapshots, this gives the storage system the opportunity to
+ // create a volume from the snapshot so that we can copy the VHD file that should be inside of the snapshot to secondary storage.
+ //
+ // The resultant volume must be writable because we need to resign the SR and the VDI that should be inside of it before we copy
+ // the VHD file to secondary storage.
+ //
+ // If the storage system is using writable snapshots, then nothing need be done by that storage system here because we can just
+ // resign the SR and the VDI that should be inside of the snapshot before copying the VHD file to secondary storage.
+ private void createVolumeFromSnapshot(HostVO hostVO, SnapshotInfo snapshotInfo, boolean keepGrantedAccess) {
+ SnapshotDetailsVO snapshotDetails = handleSnapshotDetails(snapshotInfo.getId(), "tempVolume", "create");
- copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
- }
- catch (Exception ex) {
- throw new CloudRuntimeException(ex.getMessage());
+ try {
+ snapshotInfo.getDataStore().getDriver().createAsync(snapshotInfo.getDataStore(), snapshotInfo, null);
}
finally {
- try {
- _volumeService.revokeAccess(snapshotInfo, hostVO, snapshotInfo.getDataStore());
- }
- catch (Exception ex) {
- s_logger.debug(ex.getMessage(), ex);
- }
-
- try {
- _volumeService.revokeAccess(volumeInfo, hostVO, volumeInfo.getDataStore());
- }
- catch (Exception ex) {
- s_logger.debug(ex.getMessage(), ex);
- }
+ _snapshotDetailsDao.remove(snapshotDetails.getId());
}
- String errMsg = null;
+ CopyCmdAnswer copyCmdAnswer = performResignature(snapshotInfo, hostVO, keepGrantedAccess);
if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
if (copyCmdAnswer != null && copyCmdAnswer.getDetails() != null && !copyCmdAnswer.getDetails().isEmpty()) {
- errMsg = copyCmdAnswer.getDetails();
+ throw new CloudRuntimeException(copyCmdAnswer.getDetails());
}
else {
- errMsg = "Unable to perform host-side operation";
+ throw new CloudRuntimeException("Unable to perform host-side operation");
}
}
+ }
- CopyCommandResult result = new CopyCommandResult(null, copyCmdAnswer);
+ // If the underlying storage system needed to create a volume from a snapshot for createVolumeFromSnapshot(HostVO, SnapshotInfo), then
+ // this is its opportunity to delete that temporary volume and restore properties in snapshot_details to the way they were before the
+ // invocation of createVolumeFromSnapshot(HostVO, SnapshotInfo).
+ private void deleteVolumeFromSnapshot(SnapshotInfo snapshotInfo) {
--- End diff --
@mike-tutkowski I think that use Javadoc instead of comment here might be interesting.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by mike-tutkowski <gi...@git.apache.org>.
Github user mike-tutkowski commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r53053848
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -172,6 +195,12 @@ private void validate(SnapshotInfo snapshotInfo) {
}
}
+ private boolean usingBackendSnapshotFor(SnapshotInfo snapshotInfo) {
+ String property = getProperty(snapshotInfo.getId(), "takeSnapshot");
+
+ return Boolean.parseBoolean(property);
+ }
+
private Void handleCreateTemplateFromSnapshot(SnapshotInfo snapshotInfo, TemplateInfo templateInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
--- End diff --
I believe our current best practice is to not squash commits (i.e. to preserve the history).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by swill <gi...@git.apache.org>.
Github user swill commented on the pull request:
https://github.com/apache/cloudstack/pull/1403#issuecomment-216294699
I need some code reviews here. @jburwell can you let me know what that status of your review is?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by mike-tutkowski <gi...@git.apache.org>.
Github user mike-tutkowski commented on the pull request:
https://github.com/apache/cloudstack/pull/1403#issuecomment-180977457
Thanks for the comments, Gabriel. I have pushed a new commit with those updates.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by DaanHoogland <gi...@git.apache.org>.
Github user DaanHoogland commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r62875666
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -255,99 +473,123 @@ private Void handleCreateVolumeFromSnapshotBothOnStorageSystem(SnapshotInfo snap
VolumeApiResult result = future.get();
+ if (volumeDetail != null) {
+ _volumeDetailsDao.remove(volumeDetail.getId());
+ }
+
if (result.isFailed()) {
- s_logger.debug("Failed to create a volume: " + result.getResult());
+ s_logger.warn("Failed to create a volume: " + result.getResult());
throw new CloudRuntimeException(result.getResult());
}
- }
- catch (Exception ex) {
- throw new CloudRuntimeException(ex.getMessage());
- }
- volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
+ volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
- volumeInfo.processEvent(Event.MigrationRequested);
+ volumeInfo.processEvent(Event.MigrationRequested);
- volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
+ volumeInfo = _volumeDataFactory.getVolume(volumeInfo.getId(), volumeInfo.getDataStore());
- HostVO hostVO = getHost(snapshotInfo.getDataStore().getId());
-
- String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
- int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
- CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), volumeInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+ if (useCloning) {
+ copyCmdAnswer = performResignature(volumeInfo, hostVO);
+ }
+ else {
+ // asking for a XenServer host here so we don't always prefer to use XenServer hosts that support resigning
+ // even when we don't need those hosts to do this kind of copy work
+ hostVO = getHost(snapshotInfo.getDataCenterId(), false);
- CopyCmdAnswer copyCmdAnswer = null;
+ copyCmdAnswer = performCopyOfVdi(volumeInfo, snapshotInfo, hostVO);
+ }
- try {
- _volumeService.grantAccess(snapshotInfo, hostVO, snapshotInfo.getDataStore());
- _volumeService.grantAccess(volumeInfo, hostVO, volumeInfo.getDataStore());
+ if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
+ if (copyCmdAnswer != null && !StringUtils.isEmpty(copyCmdAnswer.getDetails())) {
+ errMsg = copyCmdAnswer.getDetails();
+ }
+ else {
+ errMsg = "Unable to create volume from snapshot";
+ }
+ }
+ }
+ catch (Exception ex) {
+ errMsg = ex.getMessage() != null ? ex.getMessage() : "Copy operation failed";
+ }
- Map<String, String> srcDetails = getSnapshotDetails(_storagePoolDao.findById(snapshotInfo.getDataStore().getId()), snapshotInfo);
+ CopyCommandResult result = new CopyCommandResult(null, copyCmdAnswer);
- copyCommand.setOptions(srcDetails);
+ result.setResult(errMsg);
- Map<String, String> destDetails = getVolumeDetails(volumeInfo);
+ callback.complete(result);
+ }
- copyCommand.setOptions2(destDetails);
+ /**
+ * If the underlying storage system is making use of read-only snapshots, this gives the storage system the opportunity to
+ * create a volume from the snapshot so that we can copy the VHD file that should be inside of the snapshot to secondary storage.
+ *
+ * The resultant volume must be writable because we need to resign the SR and the VDI that should be inside of it before we copy
+ * the VHD file to secondary storage.
+ *
+ * If the storage system is using writable snapshots, then nothing need be done by that storage system here because we can just
+ * resign the SR and the VDI that should be inside of the snapshot before copying the VHD file to secondary storage.
+ */
+ private void createVolumeFromSnapshot(HostVO hostVO, SnapshotInfo snapshotInfo, boolean keepGrantedAccess) {
+ SnapshotDetailsVO snapshotDetails = handleSnapshotDetails(snapshotInfo.getId(), "tempVolume", "create");
- copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
- }
- catch (Exception ex) {
- throw new CloudRuntimeException(ex.getMessage());
+ try {
+ snapshotInfo.getDataStore().getDriver().createAsync(snapshotInfo.getDataStore(), snapshotInfo, null);
}
finally {
- try {
- _volumeService.revokeAccess(snapshotInfo, hostVO, snapshotInfo.getDataStore());
- }
- catch (Exception ex) {
- s_logger.debug(ex.getMessage(), ex);
- }
-
- try {
- _volumeService.revokeAccess(volumeInfo, hostVO, volumeInfo.getDataStore());
- }
- catch (Exception ex) {
- s_logger.debug(ex.getMessage(), ex);
- }
+ _snapshotDetailsDao.remove(snapshotDetails.getId());
}
- String errMsg = null;
+ CopyCmdAnswer copyCmdAnswer = performResignature(snapshotInfo, hostVO, keepGrantedAccess);
if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
- if (copyCmdAnswer != null && copyCmdAnswer.getDetails() != null && !copyCmdAnswer.getDetails().isEmpty()) {
- errMsg = copyCmdAnswer.getDetails();
+ if (copyCmdAnswer != null && !StringUtils.isEmpty(copyCmdAnswer.getDetails())) {
+ throw new CloudRuntimeException(copyCmdAnswer.getDetails());
}
else {
- errMsg = "Unable to perform host-side operation";
+ throw new CloudRuntimeException("Unable to create volume from snapshot");
}
}
+ }
- CopyCommandResult result = new CopyCommandResult(null, copyCmdAnswer);
+ /**
+ * If the underlying storage system needed to create a volume from a snapshot for createVolumeFromSnapshot(HostVO, SnapshotInfo), then
+ * this is its opportunity to delete that temporary volume and restore properties in snapshot_details to the way they were before the
--- End diff --
isn't the opportunity the at the caller of this method instead of in here?
I don't see this javadoc as an added value here, especially as it is on a private method. It seems more of a maintenance burden.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by mike-tutkowski <gi...@git.apache.org>.
Github user mike-tutkowski commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r63131303
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -132,31 +171,52 @@ public void copyAsync(DataObject srcData, DataObject destData, Host destHost, As
validate(snapshotInfo);
- boolean canHandleSrc = canHandle(srcData.getDataStore());
+ boolean canHandleSrc = canHandle(srcData);
if (canHandleSrc && destData instanceof TemplateInfo &&
(destData.getDataStore().getRole() == DataStoreRole.Image || destData.getDataStore().getRole() == DataStoreRole.ImageCache)) {
handleCreateTemplateFromSnapshot(snapshotInfo, (TemplateInfo)destData, callback);
+
return;
}
if (destData instanceof VolumeInfo) {
VolumeInfo volumeInfo = (VolumeInfo)destData;
- boolean canHandleDest = canHandle(destData.getDataStore());
+
+ boolean canHandleDest = canHandle(destData);
if (canHandleSrc && canHandleDest) {
- handleCreateVolumeFromSnapshotBothOnStorageSystem(snapshotInfo, volumeInfo, callback);
- return;
+ if (snapshotInfo.getDataStore().getId() == volumeInfo.getDataStore().getId()) {
+ handleCreateVolumeFromSnapshotBothOnStorageSystem(snapshotInfo, volumeInfo, callback);
+ return;
+ }
+ else {
+ throw new UnsupportedOperationException("This operation is not supported (DataStoreCapabilities.STORAGE_SYSTEM_SNAPSHOT " +
+ "not supported by source or destination storage plug-in).");
+ }
}
+
if (canHandleSrc) {
throw new UnsupportedOperationException("This operation is not supported (DataStoreCapabilities.STORAGE_SYSTEM_SNAPSHOT " +
"not supported by destination storage plug-in).");
}
+
if (canHandleDest) {
throw new UnsupportedOperationException("This operation is not supported (DataStoreCapabilities.STORAGE_SYSTEM_SNAPSHOT " +
"not supported by source storage plug-in).");
}
}
+ } else if (srcData instanceof TemplateInfo && destData instanceof VolumeInfo) {
+ boolean canHandleSrc = canHandle(srcData);
+
+ if (!canHandleSrc) {
+ throw new UnsupportedOperationException("This operation is not supported (DataStoreCapabilities.STORAGE_CAN_CREATE_VOLUME_FROM_VOLUME " +
--- End diff --
Done
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by mike-tutkowski <gi...@git.apache.org>.
Github user mike-tutkowski commented on the pull request:
https://github.com/apache/cloudstack/pull/1403#issuecomment-218869567
@DaanHoogland I think I addressed all your concerns. I plan to rebuild this locally, then push to GitHub in a bit.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by DaanHoogland <gi...@git.apache.org>.
Github user DaanHoogland commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r62747336
--- Diff: core/src/com/cloud/agent/api/StartupRoutingCommand.java ---
@@ -35,7 +35,7 @@
long memory;
long dom0MinMemory;
boolean poolSync;
-
+ private boolean _supportsClonedVolumes;
--- End diff --
I agree with Rafael on this. Let's not introduce more members starting with underscore but grow to cleaner code (slowly)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r63108616
--- Diff: engine/schema/src/com/cloud/dc/dao/ClusterDaoImpl.java ---
@@ -260,4 +268,41 @@ public boolean remove(Long id) {
sc.setParameters("dataCenterId", zoneId);
return customSearch(sc, null);
}
+
+ @Override
+ public boolean computeWhetherClusterSupportsResigning(long clusterId) {
+ ClusterVO cluster = findById(clusterId);
+
+ if (cluster == null || cluster.getAllocationState() != Grouping.AllocationState.Enabled) {
+ return false;
+ }
+
+ List<HostVO> hosts = hostDao.findByClusterId(clusterId);
+
+ if (hosts == null) {
+ return false;
+ }
+
+ for (HostVO host : hosts) {
+ if (host == null) {
+ return false;
+ }
+
+ DetailVO hostDetail = hostDetailsDao.findDetail(host.getId(), "supportsResign");
+
+ if (hostDetail == null) {
+ return false;
+ }
+
+ String value = hostDetail.getValue();
+
+ Boolean booleanValue = Boolean.valueOf(value);
+
+ if (booleanValue == false) {
+ return false;
+ }
+ }
+
+ return true;
+ }
--- End diff --
The DAO calls in the ``for`` loop are concerning from a performance perspective (both for the speed of this call and the load placed on the database). Ideally, it would be calculated in a single join. Would it be possible to craft a query across the details for all hosts in a cluster where the detail key = "supportsResign" and the value = "true"?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r63112646
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -361,59 +608,225 @@ private String getProperty(long snapshotId, String property) {
}
private Map<String, String> getVolumeDetails(VolumeInfo volumeInfo) {
- Map<String, String> sourceDetails = new HashMap<String, String>();
+ Map<String, String> volumeDetails = new HashMap<String, String>();
VolumeVO volumeVO = _volumeDao.findById(volumeInfo.getId());
long storagePoolId = volumeVO.getPoolId();
StoragePoolVO storagePoolVO = _storagePoolDao.findById(storagePoolId);
- sourceDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
- sourceDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
- sourceDetails.put(DiskTO.IQN, volumeVO.get_iScsiName());
+ volumeDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
+ volumeDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
+ volumeDetails.put(DiskTO.IQN, volumeVO.get_iScsiName());
ChapInfo chapInfo = _volumeService.getChapInfo(volumeInfo, volumeInfo.getDataStore());
if (chapInfo != null) {
- sourceDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, chapInfo.getInitiatorUsername());
- sourceDetails.put(DiskTO.CHAP_INITIATOR_SECRET, chapInfo.getInitiatorSecret());
- sourceDetails.put(DiskTO.CHAP_TARGET_USERNAME, chapInfo.getTargetUsername());
- sourceDetails.put(DiskTO.CHAP_TARGET_SECRET, chapInfo.getTargetSecret());
+ volumeDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, chapInfo.getInitiatorUsername());
+ volumeDetails.put(DiskTO.CHAP_INITIATOR_SECRET, chapInfo.getInitiatorSecret());
+ volumeDetails.put(DiskTO.CHAP_TARGET_USERNAME, chapInfo.getTargetUsername());
+ volumeDetails.put(DiskTO.CHAP_TARGET_SECRET, chapInfo.getTargetSecret());
+ }
+
+ return volumeDetails;
+ }
+
+ private Map<String, String> getSnapshotDetails(SnapshotInfo snapshotInfo) {
+ Map<String, String> snapshotDetails = new HashMap<String, String>();
+
+ long storagePoolId = snapshotInfo.getDataStore().getId();
+ StoragePoolVO storagePoolVO = _storagePoolDao.findById(storagePoolId);
+
+ snapshotDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
+ snapshotDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
+
+ long snapshotId = snapshotInfo.getId();
+
+ snapshotDetails.put(DiskTO.IQN, getProperty(snapshotId, DiskTO.IQN));
+
+ snapshotDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, getProperty(snapshotId, DiskTO.CHAP_INITIATOR_USERNAME));
+ snapshotDetails.put(DiskTO.CHAP_INITIATOR_SECRET, getProperty(snapshotId, DiskTO.CHAP_INITIATOR_SECRET));
+ snapshotDetails.put(DiskTO.CHAP_TARGET_USERNAME, getProperty(snapshotId, DiskTO.CHAP_TARGET_USERNAME));
+ snapshotDetails.put(DiskTO.CHAP_TARGET_SECRET, getProperty(snapshotId, DiskTO.CHAP_TARGET_SECRET));
+
+ return snapshotDetails;
+ }
+
+ private HostVO getHost(SnapshotInfo snapshotInfo) {
+ HostVO hostVO = getHost(snapshotInfo.getDataCenterId(), true);
+
+ if (hostVO == null) {
+ hostVO = getHost(snapshotInfo.getDataCenterId(), false);
+
+ if (hostVO == null) {
+ throw new CloudRuntimeException("Unable to locate an applicable host");
+ }
}
- return sourceDetails;
+ return hostVO;
}
- public HostVO getHost(long dataStoreId) {
- StoragePoolVO storagePoolVO = _storagePoolDao.findById(dataStoreId);
+ private HostVO getHost(Long zoneId, boolean computeClusterMustSupportResign) {
+ if (zoneId == null) {
+ throw new CloudRuntimeException("Zone ID cannot be null.");
+ }
- List<? extends Cluster> clusters = _mgr.searchForClusters(storagePoolVO.getDataCenterId(), new Long(0), Long.MAX_VALUE, HypervisorType.XenServer.toString());
+ List<? extends Cluster> clusters = _mgr.searchForClusters(zoneId, new Long(0), Long.MAX_VALUE, HypervisorType.XenServer.toString());
if (clusters == null) {
- throw new CloudRuntimeException("Unable to locate an applicable cluster");
+ clusters = new ArrayList<>();
}
+ Collections.shuffle(clusters, new Random(System.nanoTime()));
+
+ clusters:
for (Cluster cluster : clusters) {
if (cluster.getAllocationState() == AllocationState.Enabled) {
List<HostVO> hosts = _hostDao.findByClusterId(cluster.getId());
if (hosts != null) {
+ Collections.shuffle(hosts, new Random(System.nanoTime()));
+
for (HostVO host : hosts) {
if (host.getResourceState() == ResourceState.Enabled) {
- return host;
+ if (computeClusterMustSupportResign) {
+ if (computeClusterSupportsResign(cluster.getId())) {
+ return host;
+ }
+ else {
+ // no other host in the cluster in question should be able to satisfy our requirements here, so move on to the next cluster
+ continue clusters;
+ }
+ }
+ else {
+ return host;
+ }
}
}
}
}
}
- throw new CloudRuntimeException("Unable to locate an applicable cluster");
+ return null;
}
@Override
public void copyAsync(Map<VolumeInfo, DataStore> volumeMap, VirtualMachineTO vmTo, Host srcHost, Host destHost, AsyncCompletionCallback<CopyCommandResult> callback) {
CopyCommandResult result = new CopyCommandResult(null, null);
+
result.setResult("Unsupported operation requested for copying data.");
+
callback.complete(result);
}
+
+ private Map<String, String> getDetails(DataObject dataObj) {
+ if (dataObj instanceof VolumeInfo) {
+ return getVolumeDetails((VolumeInfo)dataObj);
+ }
+ else if (dataObj instanceof SnapshotInfo) {
+ return getSnapshotDetails((SnapshotInfo)dataObj);
+ }
--- End diff --
@DaanHoogland I have thoughts, but need to review the codebase before making a well-informed proposal. I will try to get my thoughts together for dev@ in the near future.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r63111740
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -172,78 +258,207 @@ private void validate(SnapshotInfo snapshotInfo) {
}
}
- private Void handleCreateTemplateFromSnapshot(SnapshotInfo snapshotInfo, TemplateInfo templateInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
+ private boolean usingBackendSnapshotFor(SnapshotInfo snapshotInfo) {
+ String property = getProperty(snapshotInfo.getId(), "takeSnapshot");
+
+ return Boolean.parseBoolean(property);
+ }
+
+ private void handleCreateTemplateFromSnapshot(SnapshotInfo snapshotInfo, TemplateInfo templateInfo, AsyncCompletionCallback<CopyCommandResult> callback) {
try {
snapshotInfo.processEvent(Event.CopyingRequested);
}
catch (Exception ex) {
throw new CloudRuntimeException("This snapshot is not currently in a state where it can be used to create a template.");
}
- HostVO hostVO = getHost(snapshotInfo.getDataStore().getId());
- DataStore srcDataStore = snapshotInfo.getDataStore();
+ HostVO hostVO = getHost(snapshotInfo);
- String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
- int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
- CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), templateInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+ boolean usingBackendSnapshot = usingBackendSnapshotFor(snapshotInfo);
+ boolean computeClusterSupportsResign = clusterDao.computeWhetherClusterSupportsResigning(hostVO.getClusterId());
- String errMsg = null;
-
- CopyCmdAnswer copyCmdAnswer = null;
+ if (usingBackendSnapshot && !computeClusterSupportsResign) {
+ throw new CloudRuntimeException("Unable to locate an applicable host with which to perform a resignature operation");
+ }
try {
- _volumeService.grantAccess(snapshotInfo, hostVO, srcDataStore);
+ if (usingBackendSnapshot) {
+ createVolumeFromSnapshot(hostVO, snapshotInfo, true);
+ }
- Map<String, String> srcDetails = getSnapshotDetails(_storagePoolDao.findById(srcDataStore.getId()), snapshotInfo);
+ DataStore srcDataStore = snapshotInfo.getDataStore();
- copyCommand.setOptions(srcDetails);
+ String value = _configDao.getValue(Config.PrimaryStorageDownloadWait.toString());
+ int primaryStorageDownloadWait = NumbersUtil.parseInt(value, Integer.parseInt(Config.PrimaryStorageDownloadWait.getDefaultValue()));
+ CopyCommand copyCommand = new CopyCommand(snapshotInfo.getTO(), templateInfo.getTO(), primaryStorageDownloadWait, VirtualMachineManager.ExecuteInSequence.value());
+
+ String errMsg = null;
+
+ CopyCmdAnswer copyCmdAnswer = null;
- copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
- }
- catch (Exception ex) {
- throw new CloudRuntimeException(ex.getMessage());
- }
- finally {
try {
- _volumeService.revokeAccess(snapshotInfo, hostVO, srcDataStore);
+ // If we are using a back-end snapshot, then we should still have access to it from the hosts in the cluster that hostVO is in
+ // (because we passed in true as the third parameter to createVolumeFromSnapshot above).
+ if (usingBackendSnapshot == false) {
+ _volumeService.grantAccess(snapshotInfo, hostVO, srcDataStore);
+ }
+
+ Map<String, String> srcDetails = getSnapshotDetails(snapshotInfo);
+
+ copyCommand.setOptions(srcDetails);
+
+ copyCmdAnswer = (CopyCmdAnswer)_agentMgr.send(hostVO.getId(), copyCommand);
}
- catch (Exception ex) {
- s_logger.debug(ex.getMessage(), ex);
+ catch (AgentUnavailableException | OperationTimedoutException ex) {
+ throw new CloudRuntimeException("Failed to create template from snapshot : " + ex.getMessage());
}
+ finally {
+ try {
+ _volumeService.revokeAccess(snapshotInfo, hostVO, srcDataStore);
+ }
+ catch (Exception ex) {
+ s_logger.warn("error during revoking access to snapshot: " + ex.getMessage(), ex);
+ }
- if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
- if (copyCmdAnswer != null && copyCmdAnswer.getDetails() != null && !copyCmdAnswer.getDetails().isEmpty()) {
- errMsg = copyCmdAnswer.getDetails();
+ if (copyCmdAnswer == null || !copyCmdAnswer.getResult()) {
+ if (copyCmdAnswer != null && !StringUtils.isEmpty(copyCmdAnswer.getDetails())) {
+ errMsg = copyCmdAnswer.getDetails();
+ }
+ else {
+ errMsg = "Unable to create template from snapshot";
--- End diff --
Please include the snapshot id in the error message to assist operational debugging.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r63112751
--- Diff: engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java ---
@@ -361,59 +600,205 @@ private String getProperty(long snapshotId, String property) {
}
private Map<String, String> getVolumeDetails(VolumeInfo volumeInfo) {
- Map<String, String> sourceDetails = new HashMap<String, String>();
+ Map<String, String> volumeDetails = new HashMap<String, String>();
VolumeVO volumeVO = _volumeDao.findById(volumeInfo.getId());
long storagePoolId = volumeVO.getPoolId();
StoragePoolVO storagePoolVO = _storagePoolDao.findById(storagePoolId);
- sourceDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
- sourceDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
- sourceDetails.put(DiskTO.IQN, volumeVO.get_iScsiName());
+ volumeDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
+ volumeDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
+ volumeDetails.put(DiskTO.IQN, volumeVO.get_iScsiName());
ChapInfo chapInfo = _volumeService.getChapInfo(volumeInfo, volumeInfo.getDataStore());
if (chapInfo != null) {
- sourceDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, chapInfo.getInitiatorUsername());
- sourceDetails.put(DiskTO.CHAP_INITIATOR_SECRET, chapInfo.getInitiatorSecret());
- sourceDetails.put(DiskTO.CHAP_TARGET_USERNAME, chapInfo.getTargetUsername());
- sourceDetails.put(DiskTO.CHAP_TARGET_SECRET, chapInfo.getTargetSecret());
+ volumeDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, chapInfo.getInitiatorUsername());
+ volumeDetails.put(DiskTO.CHAP_INITIATOR_SECRET, chapInfo.getInitiatorSecret());
+ volumeDetails.put(DiskTO.CHAP_TARGET_USERNAME, chapInfo.getTargetUsername());
+ volumeDetails.put(DiskTO.CHAP_TARGET_SECRET, chapInfo.getTargetSecret());
}
- return sourceDetails;
+ return volumeDetails;
}
- public HostVO getHost(long dataStoreId) {
- StoragePoolVO storagePoolVO = _storagePoolDao.findById(dataStoreId);
+ private Map<String, String> getSnapshotDetails(SnapshotInfo snapshotInfo) {
+ Map<String, String> snapshotDetails = new HashMap<String, String>();
+
+ long storagePoolId = snapshotInfo.getDataStore().getId();
+ StoragePoolVO storagePoolVO = _storagePoolDao.findById(storagePoolId);
+
+ snapshotDetails.put(DiskTO.STORAGE_HOST, storagePoolVO.getHostAddress());
+ snapshotDetails.put(DiskTO.STORAGE_PORT, String.valueOf(storagePoolVO.getPort()));
+
+ long snapshotId = snapshotInfo.getId();
+
+ snapshotDetails.put(DiskTO.IQN, getProperty(snapshotId, DiskTO.IQN));
- List<? extends Cluster> clusters = _mgr.searchForClusters(storagePoolVO.getDataCenterId(), new Long(0), Long.MAX_VALUE, HypervisorType.XenServer.toString());
+ snapshotDetails.put(DiskTO.CHAP_INITIATOR_USERNAME, getProperty(snapshotId, DiskTO.CHAP_INITIATOR_USERNAME));
+ snapshotDetails.put(DiskTO.CHAP_INITIATOR_SECRET, getProperty(snapshotId, DiskTO.CHAP_INITIATOR_SECRET));
+ snapshotDetails.put(DiskTO.CHAP_TARGET_USERNAME, getProperty(snapshotId, DiskTO.CHAP_TARGET_USERNAME));
+ snapshotDetails.put(DiskTO.CHAP_TARGET_SECRET, getProperty(snapshotId, DiskTO.CHAP_TARGET_SECRET));
- if (clusters == null) {
- throw new CloudRuntimeException("Unable to locate an applicable cluster");
+ return snapshotDetails;
+ }
+
+ private HostVO getHost(SnapshotInfo snapshotInfo) {
+ HostVO hostVO = getHost(snapshotInfo.getDataCenterId(), true);
+
+ if (hostVO == null) {
+ hostVO = getHost(snapshotInfo.getDataCenterId(), false);
+
+ if (hostVO == null) {
+ throw new CloudRuntimeException("Unable to locate an applicable host in data center with ID = " + snapshotInfo.getDataCenterId());
+ }
}
- for (Cluster cluster : clusters) {
- if (cluster.getAllocationState() == AllocationState.Enabled) {
- List<HostVO> hosts = _hostDao.findByClusterId(cluster.getId());
+ return hostVO;
+ }
+
+ private HostVO getHost(Long zoneId, boolean computeClusterMustSupportResign) {
+ Preconditions.checkArgument(zoneId != null, "Zone ID cannot be null.");
- if (hosts != null) {
- for (HostVO host : hosts) {
- if (host.getResourceState() == ResourceState.Enabled) {
- return host;
- }
- }
+ List<HostVO> hosts = _hostDao.listByDataCenterIdAndHypervisorType(zoneId, HypervisorType.XenServer);
+
+ if (hosts == null) {
+ return null;
+ }
+
+ List<Long> clustersToSkip = new ArrayList<>();
+
+ Collections.shuffle(hosts, random);
+
+ for (HostVO host : hosts) {
+ if (computeClusterMustSupportResign) {
+ long clusterId = host.getClusterId();
+
+ if (clustersToSkip.contains(clusterId)) {
+ continue;
+ }
+
+ if (clusterDao.computeWhetherClusterSupportsResigning(clusterId)) {
+ return host;
+ }
+ else {
+ clustersToSkip.add(clusterId);
}
}
+ else {
+ return host;
+ }
}
- throw new CloudRuntimeException("Unable to locate an applicable cluster");
+ return null;
}
@Override
public void copyAsync(Map<VolumeInfo, DataStore> volumeMap, VirtualMachineTO vmTo, Host srcHost, Host destHost, AsyncCompletionCallback<CopyCommandResult> callback) {
CopyCommandResult result = new CopyCommandResult(null, null);
+
result.setResult("Unsupported operation requested for copying data.");
+
callback.complete(result);
}
+
+ private Map<String, String> getDetails(DataObject dataObj) {
+ if (dataObj instanceof VolumeInfo) {
+ return getVolumeDetails((VolumeInfo)dataObj);
+ }
+ else if (dataObj instanceof SnapshotInfo) {
+ return getSnapshotDetails((SnapshotInfo)dataObj);
+ }
+
+ throw new CloudRuntimeException("'dataObj' must be of type 'VolumeInfo' or 'SnapshotInfo'.");
+ }
+
+ private CopyCmdAnswer performResignature(DataObject dataObj, HostVO hostVO) {
+ return performResignature(dataObj, hostVO, false);
+ }
+
+ private CopyCmdAnswer performResignature(DataObject dataObj, HostVO hostVO, boolean keepGrantedAccess) {
+ long storagePoolId = dataObj.getDataStore().getId();
+ DataStore dataStore = dataStoreMgr.getDataStore(storagePoolId, DataStoreRole.Primary);
+
+ Map<String, String> details = getDetails(dataObj);
+
+ ResignatureCommand command = new ResignatureCommand(details);
+
+ ResignatureAnswer answer = null;
+
+ try {
+ _volumeService.grantAccess(dataObj, hostVO, dataStore);
+
+ answer = (ResignatureAnswer)_agentMgr.send(hostVO.getId(), command);
+ }
+ catch (AgentUnavailableException | OperationTimedoutException ex) {
+ keepGrantedAccess = false;
+
+ String msg = "Failed to revoke access to the volume from the hosts : ";
+
+ s_logger.warn(msg + ex.getMessage());
--- End diff --
Why not ``s_logger.warn(msg, ex)`` to log the entire stack trace?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Taking fast and efficient volume snapshot...
Posted by DaanHoogland <gi...@git.apache.org>.
Github user DaanHoogland commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1403#discussion_r62890766
--- Diff: plugins/storage/volume/solidfire/src/org/apache/cloudstack/storage/datastore/driver/SolidFirePrimaryDataStoreDriver.java ---
@@ -665,22 +950,190 @@ private void updateSnapshotDetails(long csSnapshotId, long sfNewVolumeId, long s
_snapshotDetailsDao.persist(snapshotDetail);
}
- // return null for no error message
- private String deleteSnapshot(SnapshotInfo snapshotInfo, long storagePoolId) {
- String errMsg = null;
+ private String createVolume(VolumeInfo volumeInfo, long storagePoolId) {
+ verifySufficientBytesForStoragePool(volumeInfo, storagePoolId);
+ verifySufficientIopsForStoragePool(volumeInfo.getMinIops() != null ? volumeInfo.getMinIops() : getDefaultMinIops(storagePoolId), storagePoolId);
+
+ SolidFireUtil.SolidFireConnection sfConnection = SolidFireUtil.getSolidFireConnection(storagePoolId, _storagePoolDetailsDao);
+
+ long sfAccountId = getCreateSolidFireAccountId(sfConnection, volumeInfo.getAccountId(), storagePoolId);
+
+ long csSnapshotId = getCsIdForCloning(volumeInfo.getId(), "cloneOfSnapshot");
+ long csTemplateId = getCsIdForCloning(volumeInfo.getId(), "cloneOfTemplate");
+
+ SolidFireUtil.SolidFireVolume sfVolume;
+
+ if (csSnapshotId > 0) {
+ // We are supposed to create a clone of the underlying volume or snapshot that supports the CloudStack snapshot.
+ sfVolume = createClone(sfConnection, csSnapshotId, volumeInfo, sfAccountId, storagePoolId, DataObjectType.SNAPSHOT);
+ } else if (csTemplateId > 0) {
+ // Clone from template.
+ sfVolume = createClone(sfConnection, csTemplateId, volumeInfo, sfAccountId, storagePoolId, DataObjectType.TEMPLATE);
+
+ long volumeSize = getDataObjectSizeIncludingHypervisorSnapshotReserve(volumeInfo, _storagePoolDao.findById(storagePoolId));
+
+ if (volumeSize > sfVolume.getTotalSize()) {
+ // Expand the volume to include HSR.
+ SolidFireUtil.modifySolidFireVolume(sfConnection, sfVolume.getId(), volumeSize, getVolumeAttributes(volumeInfo),
+ sfVolume.getMinIops(), sfVolume.getMaxIops(), sfVolume.getBurstIops());
+
+ // Get the SolidFire volume from the SAN again because we just updated its size.
+ sfVolume = SolidFireUtil.getSolidFireVolume(sfConnection, sfVolume.getId());
+ }
+ }
+ else {
+ sfVolume = createSolidFireVolume(sfConnection, volumeInfo, sfAccountId);
+ }
+
+ String iqn = sfVolume.getIqn();
+
+ VolumeVO volume = _volumeDao.findById(volumeInfo.getId());
+
+ volume.set_iScsiName(iqn);
+ volume.setFolder(String.valueOf(sfVolume.getId()));
+ volume.setPoolType(StoragePoolType.IscsiLUN);
+ volume.setPoolId(storagePoolId);
+
+ _volumeDao.update(volume.getId(), volume);
+
+ updateVolumeDetails(volume.getId(), sfVolume.getTotalSize());
+
+ StoragePoolVO storagePool = _storagePoolDao.findById(storagePoolId);
+
+ long capacityBytes = storagePool.getCapacityBytes();
+ // getUsedBytes(StoragePool) will include the bytes of the newly created volume because
+ // updateVolumeDetails(long, long) has already been called for this volume
+ long usedBytes = getUsedBytes(storagePool);
+
+ storagePool.setUsedBytes(usedBytes > capacityBytes ? capacityBytes : usedBytes);
+
+ _storagePoolDao.update(storagePoolId, storagePool);
+
+ return iqn;
+ }
+
+ private void createTempVolume(SnapshotInfo snapshotInfo, long storagePoolId) {
+ long csSnapshotId = snapshotInfo.getId();
+
+ SnapshotDetailsVO snapshotDetails = _snapshotDetailsDao.findDetail(csSnapshotId, SolidFireUtil.SNAPSHOT_ID);
+
+ if (snapshotDetails == null || snapshotDetails.getValue() == null) {
+ throw new CloudRuntimeException("'createTempVolume(SnapshotInfo, long)' should not be invoked unless " + SolidFireUtil.SNAPSHOT_ID + " exists.");
+ }
+
+ SolidFireUtil.SolidFireConnection sfConnection = SolidFireUtil.getSolidFireConnection(storagePoolId, _storagePoolDetailsDao);
+
+ snapshotDetails = _snapshotDetailsDao.findDetail(csSnapshotId, "tempVolume");
+
+ if (snapshotDetails != null && snapshotDetails.getValue() != null && snapshotDetails.getValue().equalsIgnoreCase("create")) {
+ long sfAccountId = getCreateSolidFireAccountId(sfConnection, snapshotInfo.getAccountId(), storagePoolId);
+
+ SolidFireUtil.SolidFireVolume sfVolume = createCloneFromSnapshot(sfConnection, csSnapshotId, sfAccountId);
+
+ addTempVolumeId(csSnapshotId, String.valueOf(sfVolume.getId()));
+
+ handleSnapshotDetails(csSnapshotId, DiskTO.IQN, sfVolume.getIqn());
+ }
+ else if (snapshotDetails != null && snapshotDetails.getValue() != null && snapshotDetails.getValue().equalsIgnoreCase("delete")) {
+ snapshotDetails = _snapshotDetailsDao.findDetail(csSnapshotId, SolidFireUtil.VOLUME_ID);
+
+ SolidFireUtil.deleteSolidFireVolume(sfConnection, Long.parseLong(snapshotDetails.getValue()));
+
+ removeTempVolumeId(csSnapshotId);
+
+ snapshotDetails = _snapshotDetailsDao.findDetail(csSnapshotId, DiskTO.IQN);
+
+ _snapshotDetailsDao.remove(snapshotDetails.getId());
+ }
+ else {
+ throw new CloudRuntimeException("Invalid state in 'createTempVolume(SnapshotInfo, long)'");
+ }
+ }
+
+ private String createTemplateVolume(TemplateInfo templateInfo, long storagePoolId) {
+ verifySufficientBytesForStoragePool(templateInfo, storagePoolId);
+
+ SolidFireUtil.SolidFireConnection sfConnection = SolidFireUtil.getSolidFireConnection(storagePoolId, _storagePoolDetailsDao);
+
+ long sfAccountId = getCreateSolidFireAccountId(sfConnection, templateInfo.getAccountId(), storagePoolId);
+
+ SolidFireUtil.SolidFireVolume sfVolume = createSolidFireVolume(sfConnection, templateInfo, sfAccountId);
+
+ String iqn = sfVolume.getIqn();
+
+ VMTemplateStoragePoolVO templatePoolRef = _tmpltPoolDao.findByPoolTemplate(storagePoolId, templateInfo.getId());
+
+ templatePoolRef.setInstallPath(iqn);
+ templatePoolRef.setLocalDownloadPath(Long.toString(sfVolume.getId()));
+ templatePoolRef.setTemplateSize(sfVolume.getTotalSize());
+
+ _tmpltPoolDao.update(templatePoolRef.getId(), templatePoolRef);
+
+ StoragePoolVO storagePool = _storagePoolDao.findById(storagePoolId);
+
+ long capacityBytes = storagePool.getCapacityBytes();
+ // getUsedBytes(StoragePool) will include the bytes of the newly created template volume because
+ // _tmpltPoolDao.update(Long, VMTemplateStoragePoolVO) has already been invoked
+ long usedBytes = getUsedBytes(storagePool);
+
+ storagePool.setUsedBytes(usedBytes > capacityBytes ? capacityBytes : usedBytes);
+
+ _storagePoolDao.update(storagePoolId, storagePool);
+
+ return iqn;
+ }
+
+ private void deleteVolume(VolumeInfo volumeInfo, long storagePoolId) {
+ try {
+ long volumeId = volumeInfo.getId();
+
+ SolidFireUtil.SolidFireConnection sfConnection = SolidFireUtil.getSolidFireConnection(storagePoolId, _storagePoolDetailsDao);
+
+ deleteSolidFireVolume(sfConnection, volumeInfo);
+
+ _volumeDetailsDao.removeDetails(volumeId);
+
+ StoragePoolVO storagePool = _storagePoolDao.findById(storagePoolId);
+
+ long usedBytes = getUsedBytes(storagePool, volumeId);
+
+ storagePool.setUsedBytes(usedBytes < 0 ? 0 : usedBytes);
+
+ _storagePoolDao.update(storagePoolId, storagePool);
+ }
+ catch (Exception ex) {
--- End diff --
needed here? and why?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---