You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by rhtyd <gi...@git.apache.org> on 2016/05/13 03:27:28 UTC
[GitHub] cloudstack pull request: CLOUDSTACK-9348: Make NioConnectio loop l...
GitHub user rhtyd opened a pull request:
https://github.com/apache/cloudstack/pull/1543
CLOUDSTACK-9348: Make NioConnectio loop less aggressive
- Reverts ea2286 that introduced a wakeup on each connection loop run.
- In SSL handshake code removes delegated tasks to be run in separate threads.
/cc @kiwiflyer @swill @jburwell and others for review
@kiwiflyer please help me test this fix and share if it makes the NioConnection robust now, without having the selector consume a lot of CPU. Thanks.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/shapeblue/cloudstack nio-fix-aggressive-cpu-use
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/cloudstack/pull/1543.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1543
----
commit 540d9572fd491db3ce182d26636fc74ada4e171c
Author: Rohit Yadav <ro...@shapeblue.com>
Date: 2016-05-13T03:24:24Z
CLOUDSTACK-9348: Make NioConnectio loop less aggressive
- Reverts ea2286 that introduced a wakeup on each connection loop run.
- In SSL handshake code removes delegated tasks to be run in separate threads.
Signed-off-by: Rohit Yadav <ro...@shapeblue.com>
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Fix Nio/CPU issue and CI failures
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on the pull request:
https://github.com/apache/cloudstack/pull/1543#issuecomment-219095201
Thanks @swill
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9348: Make NioConnectio loop l...
Posted by kiwiflyer <gi...@git.apache.org>.
Github user kiwiflyer commented on the pull request:
https://github.com/apache/cloudstack/pull/1543#issuecomment-218947359
@rhtyd This patch seems to have fixed the cpu issue. I've got this running on a hardware lab right now.
We'll also install this on a larger lab tomorrow and stress test the agent a bit.
Thought you'd like to know before I hit the sack for the night!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9348: Make NioConnectio loop l...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on the pull request:
https://github.com/apache/cloudstack/pull/1543#issuecomment-218947733
@kiwiflyer Thanks you for testing this so quickly and sharing.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9348: Make NioConnectio loop l...
Posted by swill <gi...@git.apache.org>.
Github user swill commented on the pull request:
https://github.com/apache/cloudstack/pull/1543#issuecomment-218947527
Thanks for the fix @rhtyd and for testing this @kiwiflyer. \U0001f44d
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Fix Nio/CPU issue and CI failures
Posted by DaanHoogland <gi...@git.apache.org>.
Github user DaanHoogland commented on the pull request:
https://github.com/apache/cloudstack/pull/1543#issuecomment-218980870
code LGTM. trusting @kiwiflyer I will probably test later but am a bit stuck today.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Fix Nio/CPU issue and CI failures
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/cloudstack/pull/1543
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9348: Make NioConnectio loop l...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1543#discussion_r63129136
--- Diff: utils/src/main/java/com/cloud/utils/nio/Link.java ---
@@ -615,7 +615,10 @@ public static boolean doHandshake(final SocketChannel socketChannel, final SSLEn
case NEED_TASK:
Runnable task;
while ((task = sslEngine.getDelegatedTask()) != null) {
- new Thread(task).run();
+ if (s_logger.isTraceEnabled()) {
+ s_logger.trace("SSL: Running delegated task!");
+ }
+ task.run();
--- End diff --
@jburwell thanks for pointing out, I've made the runnable task run in the current thread only. The delegated tasks in case of SSL handshake could be resolving domain names to address.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Fix Nio/CPU issue and CI failures
Posted by rhtyd <gi...@git.apache.org>.
GitHub user rhtyd reopened a pull request:
https://github.com/apache/cloudstack/pull/1543
Fix Nio/CPU issue and CI failures
- Reverts ea2286 that introduced a wakeup on each connection loop run.
- In SSL handshake code removes delegated tasks to be run in separate threads.
/cc @kiwiflyer @swill @jburwell and others for review
@kiwiflyer please help me test this fix and share if it makes the NioConnection robust now, without having the selector consume a lot of CPU. Thanks.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/shapeblue/cloudstack nio-fix-aggressive-cpu-use
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/cloudstack/pull/1543.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1543
----
commit 540d9572fd491db3ce182d26636fc74ada4e171c
Author: Rohit Yadav <ro...@shapeblue.com>
Date: 2016-05-13T03:24:24Z
CLOUDSTACK-9348: Make NioConnectio loop less aggressive
- Reverts ea2286 that introduced a wakeup on each connection loop run.
- In SSL handshake code removes delegated tasks to be run in separate threads.
Signed-off-by: Rohit Yadav <ro...@shapeblue.com>
commit ae0f169123158c23898316a32b413f4647d0c456
Author: Rohit Yadav <ro...@shapeblue.com>
Date: 2016-05-13T03:39:58Z
CLOUDSTACK-9299: Fix test failures on CI
- Fixes oobm integration test to skip if known ipmitool bug is hit
- Fixes ProcessTest unit test case to use sleep
- Removes redundant unit test that covers code in ProcessTest
Signed-off-by: Rohit Yadav <ro...@shapeblue.com>
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Fix Nio/CPU issue and CI failures
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on the pull request:
https://github.com/apache/cloudstack/pull/1543#issuecomment-218987023
I've also fixed a integration/component test that has failed on Travis several time for various PRs due to cleanup order (tearDown issue):
```
==== Marvin Init Successful ====
=== TestName: test_listVolume_as_domainadmin | Status : SUCCESS ===
=== TestName: test_listVolume_as_domainadmin_cross_domainid | Status : SUCCESS ===
=== TestName: test_listVolume_as_domainadmin_cross_domainid_accountid | Status : SUCCESS ===
=== TestName: test_listVolume_as_domainadmin_domainid | Status : SUCCESS ===
=== TestName: test_listVolume_as_domainadmin_domainid_accountid | Status : SUCCESS ===
=== TestName: test_listVolume_as_domainadmin_domainid_accountid_listall_false | Status : SUCCESS ===
=== TestName: test_listVolume_as_domainadmin_domainid_accountid_listall_false_rec_false | Status : SUCCESS ===
=== TestName: test_listVolume_as_domainadmin_domainid_accountid_listall_false_rec_true | Status : SUCCESS ===
=== TestName: test_listVolume_as_domainadmin_domainid_accountid_listall_true | Status : SUCCESS ===
=== TestName: test_listVolume_as_domainadmin_domainid_accountid_listall_true_rec_false | Status : SUCCESS ===
=== TestName: test_listVolume_as_domainadmin_domainid_accountid_listall_true_rec_true | Status : SUCCESS ===
=== TestName: test_listVolume_as_domainadmin_domainid_accountid_rec_false | Status : SUCCESS ===
=== TestName: test_listVolume_as_domainadmin_domainid_accountid_rec_true | Status : SUCCESS ===
=== TestName: test_listVolume_as_domainadmin_domainid_listall_false | Status : SUCCESS ===
=== TestName: test_listVolume_as_domainadmin_domainid_listall_false_rec_false | Status : SUCCESS ===
=== TestName: test_listVolume_as_domainadmin_domainid_listall_false_rec_true | Status : SUCCESS ===
=== TestName: test_listVolume_as_domainadmin_domainid_listall_true | Status : SUCCESS ===
=== TestName: test_listVolume_as_domainadmin_domainid_listall_true_rec_false | Status : SUCCESS ===
=== TestName: test_listVolume_as_domainadmin_domainid_listall_true_rec_true | Status : SUCCESS ===
=== TestName: test_listVolume_as_domainadmin_domainid_rec_false | Status : SUCCESS ===
=== TestName: test_listVolume_as_domainadmin_domainid_rec_true | Status : SUCCESS ===
=== TestName: test_listVolume_as_domainadmin_listall_false | Status : SUCCESS ===
=== TestName: test_listVolume_as_domainadmin_listall_false_rec_false | Status : SUCCESS ===
=== TestName: test_listVolume_as_domainadmin_listall_false_rec_true | Status : SUCCESS ===
=== TestName: test_listVolume_as_domainadmin_listall_true | Status : SUCCESS ===
=== TestName: test_listVolume_as_domainadmin_listall_true_rec_false | Status : SUCCESS ===
=== TestName: test_listVolume_as_domainadmin_listall_true_rec_true | Status : SUCCESS ===
=== TestName: test_listVolume_as_domainadmin_rec_true | Status : SUCCESS ===
=== TestName: test_listVolume_as_domainadmin_true_rec_false | Status : SUCCESS ===
=== TestName: test_listVolume_as_regularuser | Status : SUCCESS ===
=== TestName: test_listVolume_as_regularuser_cross_domainid | Status : SUCCESS ===
=== TestName: test_listVolume_as_regularuser_cross_domainid_accountid | Status : SUCCESS ===
=== TestName: test_listVolume_as_regularuser_domainid | Status : SUCCESS ===
=== TestName: test_listVolume_as_regularuser_domainid__rec_false | Status : SUCCESS ===
=== TestName: test_listVolume_as_regularuser_domainid_accountid | Status : SUCCESS ===
=== TestName: test_listVolume_as_regularuser_domainid_accountid_listall_false | Status : SUCCESS ===
=== TestName: test_listVolume_as_regularuser_domainid_accountid_listall_false_rec_false | Status : SUCCESS ===
=== TestName: test_listVolume_as_regularuser_domainid_accountid_listall_false_rec_true | Status : SUCCESS ===
=== TestName: test_listVolume_as_regularuser_domainid_accountid_listall_true | Status : SUCCESS ===
=== TestName: test_listVolume_as_regularuser_domainid_accountid_listall_true_rec_false | Status : SUCCESS ===
=== TestName: test_listVolume_as_regularuser_domainid_accountid_listall_true_rec_true | Status : SUCCESS ===
=== TestName: test_listVolume_as_regularuser_domainid_accountid_rec_false | Status : SUCCESS ===
=== TestName: test_listVolume_as_regularuser_domainid_accountid_rec_true | Status : SUCCESS ===
=== TestName: test_listVolume_as_regularuser_domainid_listall_false | Status : SUCCESS ===
=== TestName: test_listVolume_as_regularuser_domainid_listall_false_rec_false | Status : SUCCESS ===
=== TestName: test_listVolume_as_regularuser_domainid_listall_false_rec_true | Status : SUCCESS ===
=== TestName: test_listVolume_as_regularuser_domainid_listall_true | Status : SUCCESS ===
=== TestName: test_listVolume_as_regularuser_domainid_listall_true_rec_false | Status : SUCCESS ===
=== TestName: test_listVolume_as_regularuser_domainid_listall_true_rec_true | Status : SUCCESS ===
=== TestName: test_listVolume_as_regularuser_domainid_true_rec_true | Status : SUCCESS ===
=== TestName: test_listVolume_as_regularuser_listall_false | Status : SUCCESS ===
=== TestName: test_listVolume_as_regularuser_listall_false_rec_false | Status : SUCCESS ===
=== TestName: test_listVolume_as_regularuser_listall_false_rec_true | Status : SUCCESS ===
=== TestName: test_listVolume_as_regularuser_listall_true | Status : SUCCESS ===
=== TestName: test_listVolume_as_regularuser_listall_true_rec_false | Status : SUCCESS ===
=== TestName: test_listVolume_as_regularuser_listall_true_rec_true | Status : SUCCESS ===
=== TestName: test_listVolume_as_regularuser_rec_false | Status : SUCCESS ===
=== TestName: test_listVolume_as_regularuser_rec_true | Status : SUCCESS ===
=== TestName: test_listVolume_as_rootadmin | Status : SUCCESS ===
=== TestName: test_listVolume_as_rootadmin_domainid | Status : SUCCESS ===
=== TestName: test_listVolume_as_rootadmin_domainid_accountid | Status : SUCCESS ===
=== TestName: test_listVolume_as_rootadmin_domainid_accountid_listall_false | Status : SUCCESS ===
=== TestName: test_listVolume_as_rootadmin_domainid_accountid_listall_false_rec_false | Status : SUCCESS ===
=== TestName: test_listVolume_as_rootadmin_domainid_accountid_listall_false_rec_true | Status : SUCCESS ===
=== TestName: test_listVolume_as_rootadmin_domainid_accountid_listall_true | Status : SUCCESS ===
=== TestName: test_listVolume_as_rootadmin_domainid_accountid_listall_true_rec_false | Status : SUCCESS ===
=== TestName: test_listVolume_as_rootadmin_domainid_accountid_listall_true_rec_true | Status : SUCCESS ===
=== TestName: test_listVolume_as_rootadmin_domainid_accountid_rec_false | Status : SUCCESS ===
=== TestName: test_listVolume_as_rootadmin_domainid_accountid_rec_true | Status : SUCCESS ===
=== TestName: test_listVolume_as_rootadmin_domainid_listall_false | Status : SUCCESS ===
=== TestName: test_listVolume_as_rootadmin_domainid_listall_false_rec_false | Status : SUCCESS ===
=== TestName: test_listVolume_as_rootadmin_domainid_listall_false_rec_true | Status : SUCCESS ===
=== TestName: test_listVolume_as_rootadmin_domainid_listall_true | Status : SUCCESS ===
=== TestName: test_listVolume_as_rootadmin_domainid_listall_true_rec_false | Status : SUCCESS ===
=== TestName: test_listVolume_as_rootadmin_domainid_listall_true_rec_true | Status : SUCCESS ===
=== TestName: test_listVolume_as_rootadmin_domainid_rec_false | Status : SUCCESS ===
=== TestName: test_listVolume_as_rootadmin_domainid_rec_true | Status : SUCCESS ===
=== TestName: test_listVolume_as_rootadmin_listall_false | Status : SUCCESS ===
=== TestName: test_listVolume_as_rootadmin_listall_false_rec_false | Status : SUCCESS ===
=== TestName: test_listVolume_as_rootadmin_listall_false_rec_true | Status : SUCCESS ===
=== TestName: test_listVolume_as_rootadmin_listall_true | Status : SUCCESS ===
=== TestName: test_listVolume_as_rootadmin_listall_true_rec_false | Status : SUCCESS ===
=== TestName: test_listVolume_as_rootadmin_listall_true_rec_true | Status : SUCCESS ===
=== TestName: test_listVolume_as_rootadmin_rec_false | Status : SUCCESS ===
=== TestName: test_listVolume_as_rootadmin_rec_true | Status : SUCCESS ===
=== TestName: test_listVolume_by_id_as_domainadmin_ownedbyusersindomain | Status : SUCCESS ===
=== TestName: test_listVolume_by_id_as_domainadmin_ownedbyusersinsubdomain | Status : SUCCESS ===
=== TestName: test_listVolume_by_id_as_domainadmin_ownedbyusersinsubdomain2 | Status : SUCCESS ===
=== TestName: test_listVolume_by_id_as_domainadmin_ownedbyusersnotindomain | Status : SUCCESS ===
=== TestName: test_listVolume_by_id_as_domainadmin_owns | Status : SUCCESS ===
=== TestName: test_listVolume_by_id_as_rootadmin_Volumesownedbyothers | Status : SUCCESS ===
=== TestName: test_listVolume_by_id_as_rootadmin_owns | Status : SUCCESS ===
=== TestName: test_listVolume_by_id_as_user_own | Status : SUCCESS ===
=== TestName: test_listVolume_by_id_as_user_volumefromotherdomain | Status : SUCCESS ===
=== TestName: test_listVolume_by_id_as_user_volumefromsamedomaindifferentaccount | Status : SUCCESS ===
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Fix Nio/CPU issue and CI failures
Posted by swill <gi...@git.apache.org>.
Github user swill commented on the pull request:
https://github.com/apache/cloudstack/pull/1543#issuecomment-219095904
Thanks for the effort stabilizing Jenkins and Travis the last few days. This has been super helpful as we get close to the freeze. Very much appreciated. Most of the runs are coming back clean now. \U0001f44d
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Fix Nio/CPU issue and CI failures
Posted by swill <gi...@git.apache.org>.
Github user swill commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1543#discussion_r63215625
--- Diff: test/integration/component/test_acl_listvolume.py ---
@@ -357,9 +357,9 @@ def tearDownClass(cls):
cls.apiclient = super(TestVolumeList, cls).getClsTestClient().getApiClient()
cls.apiclient.connection.apiKey = cls.default_apikey
cls.apiclient.connection.securityKey = cls.default_secretkey
+ cleanup_resources(cls.apiclient, cls.cleanup)
--- End diff --
Disregard, I can't read the difference between `test_acl_listvolume.py` and `test_acl_listvm.py`. Deleting both of these comments. This post is for everyone who follows this stuff in their email and are confused...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Fix Nio/CPU issue and CI failures
Posted by abhinandanprateek <gi...@git.apache.org>.
Github user abhinandanprateek commented on the pull request:
https://github.com/apache/cloudstack/pull/1543#issuecomment-218978811
LGTM based on code review.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Fix Nio/CPU issue and CI failures
Posted by swill <gi...@git.apache.org>.
Github user swill commented on the pull request:
https://github.com/apache/cloudstack/pull/1543#issuecomment-219071379
Most of these things are fixes to Jenkins and travis issues, so those jobs being green is good validation of the fixes. The main functionality change has been validated by @kiwiflyer, so I will get this merged right away. Thanks for being on top of this @rhtyd, I really appreciate it. \U0001f44d
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Fix Nio/CPU issue and CI failures
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd closed the pull request at:
https://github.com/apache/cloudstack/pull/1543
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Fix Nio/CPU issue and CI failures
Posted by swill <gi...@git.apache.org>.
Github user swill commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1543#discussion_r63215404
--- Diff: test/integration/component/test_acl_listvolume.py ---
@@ -357,9 +357,9 @@ def tearDownClass(cls):
cls.apiclient = super(TestVolumeList, cls).getClsTestClient().getApiClient()
cls.apiclient.connection.apiKey = cls.default_apikey
cls.apiclient.connection.securityKey = cls.default_secretkey
+ cleanup_resources(cls.apiclient, cls.cleanup)
--- End diff --
Is this to fix this problem? This is from #1297, which was ran before I got this fix merged.
```
======================================================================
ERROR: test suite for <class 'integration.component.test_acl_listvm.TestVMList'>
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/travis/.local/lib/python2.7/site-packages/nose/suite.py", line 228, in run
self.tearDown()
File "/home/travis/.local/lib/python2.7/site-packages/nose/suite.py", line 351, in tearDown
self.teardownContext(ancestor)
File "/home/travis/.local/lib/python2.7/site-packages/nose/suite.py", line 367, in teardownContext
try_run(context, names)
File "/home/travis/.local/lib/python2.7/site-packages/nose/util.py", line 471, in try_run
return func()
File "/home/travis/build/apache/cloudstack/test/integration/component/test_acl_listvm.py", line 346, in tearDownClass
cls.domain_1.delete(cls.apiclient,cleanup="true")
File "/home/travis/.local/lib/python2.7/site-packages/marvin/lib/base.py", line 78, in delete
apiclient.deleteDomain(cmd)
File "/home/travis/.local/lib/python2.7/site-packages/marvin/cloudstackAPI/cloudstackAPIClient.py", line 703, in deleteDomain
response = self.connection.marvinRequest(command, response_type=response, method=method)
File "/home/travis/.local/lib/python2.7/site-packages/marvin/cloudstackConnection.py", line 379, in marvinRequest
raise e
Exception: Job failed: {jobprocstatus : 0, created : u'2016-05-13T14:25:30+0000', cmd : u'org.apache.cloudstack.api.command.admin.domain.DeleteDomainCmd', userid : u'7e18b4b2-1915-11e6-abc2-42010a800044', jobstatus : 2, jobid : u'a3e58407-3f1a-4508-9671-733029d283c0', jobresultcode : 530, jobresulttype : u'object', jobresult : {errorcode : 530, errortext : u'Failed to clean up domain resources and sub domains, delete failed on domain D1-HVOVF7 (id: 2).'}, accountid : u'7e188b32-1915-11e6-abc2-42010a800044'}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---