You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by rhtyd <gi...@git.apache.org> on 2016/05/05 17:56:01 UTC
[GitHub] cloudstack pull request: CLOUDSTACK-9348: Optimize NioTest and Nio...
GitHub user rhtyd opened a pull request:
https://github.com/apache/cloudstack/pull/1534
CLOUDSTACK-9348: Optimize NioTest and NioConnection main loop
- Reduces SSL handshake timeout to 15s, previously this was only 10s in
commit debfcdef788ce0d51be06db0ef10f6815f9b563b
- Adds an aggresive explicit wakeup to save the Nio main IO loop/handler from
getting blocked
- Fix NioTest to fail/succeed in about 60s, previously this was 300s
- Due to aggresive wakeup usage, NioTest should complete in less than 5s on most
systems. On virtualized environment this may slightly increase due to thread,
CPU burst/scheduling delays.
/cc @swill please review and merge.
Sorry about the previous values, they were not optimized for virtualized env. The aggressive selector.wakeup will ensure main IO loop does not get blocked even by malicious users, even for any timeout (ssl handshake etc).
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/shapeblue/cloudstack niotest-fix
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/cloudstack/pull/1534.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1534
----
commit ea22869593f68a3a34b12aeb23c2bb6c34efd365
Author: Rohit Yadav <ro...@shapeblue.com>
Date: 2016-05-05T17:49:33Z
CLOUDSTACK-9348: Optimize NioTest and NioConnection main loop
- Reduces SSL handshake timeout to 15s, previously this was only 10s in
commit debfcdef788ce0d51be06db0ef10f6815f9b563b
- Adds an aggresive explicit wakeup to save the Nio main IO loop/handler from
getting blocked
- Fix NioTest to fail/succeed in about 60s, previously this was 300s
- Due to aggresive wakeup usage, NioTest should complete in less than 5s on most
systems. On virtualized environment this may slightly increase due to thread,
CPU burst/scheduling delays.
Signed-off-by: Rohit Yadav <ro...@shapeblue.com>
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9348: Optimize NioTest and Nio...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on the pull request:
https://github.com/apache/cloudstack/pull/1534#issuecomment-217287468
@swill thanks for sharing, I made the NioConnection's main IO loop aggressive and reduced the SSL handshake timeout to 15s (this was previously 10s, but over last year instead of fixing core issue as we did not know that cause and details, I had increased it to 60s in Link class). This sort of optimization will help CloudStack perform speedy re-connection and handling of clients, even 1000s of malicious clients won't be able to block the main IO loop from handling other requests.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9348: Optimize NioTest and Nio...
Posted by swill <gi...@git.apache.org>.
Github user swill commented on the pull request:
https://github.com/apache/cloudstack/pull/1534#issuecomment-217521378
I am not concerned about the two failures. One happens randomly in my environment and one is a cleanup issue between test runs which is not related to this PR.
Since `master` is currently broken due to some issues with #1493, I am going to merge this right away...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9348: Optimize NioTest and Nio...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on the pull request:
https://github.com/apache/cloudstack/pull/1534#issuecomment-217442413
LGTM based on code review
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9348: Optimize NioTest and Nio...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on the pull request:
https://github.com/apache/cloudstack/pull/1534#issuecomment-217226501
@DaanHoogland @jburwell @wido @swill and others -- please review, this mainly fixes the NioTest which was failing so if it's okay and works for Travis and Will's CI let's merge this. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9348: Optimize NioTest and Nio...
Posted by swill <gi...@git.apache.org>.
Github user swill commented on the pull request:
https://github.com/apache/cloudstack/pull/1534#issuecomment-217278185
Well the test runs a lot faster/cleaner now. \U0001f44d
```
Running com.cloud.utils.testcase.NioTest
2016-05-05 22:53:54,828 INFO [utils.testcase.NioTest] (main:) Setting up Benchmark Test
2016-05-05 22:53:54,861 INFO [utils.nio.NioServer] (main:) NioConnection started and listening on /0:0:0:0:0:0:0:0:41317
2016-05-05 22:53:54,874 INFO [utils.testcase.NioTest] (MaliciousNioClientHandler-1:) Connecting to 127.0.0.1:41317
2016-05-05 22:53:54,874 INFO [utils.testcase.NioTest] (MaliciousNioClientHandler-3:) Connecting to 127.0.0.1:41317
2016-05-05 22:53:54,874 INFO [utils.testcase.NioTest] (MaliciousNioClientHandler-4:) Connecting to 127.0.0.1:41317
2016-05-05 22:53:54,882 DEBUG [utils.testcase.NioTest] (Time-limited test:) 0/4 tests done. Waiting for completion
2016-05-05 22:53:54,875 INFO [utils.testcase.NioTest] (MaliciousNioClientHandler-2:) Connecting to 127.0.0.1:41317
2016-05-05 22:53:54,885 INFO [utils.nio.NioClient] (NioClientHandler-4:) Connecting to 127.0.0.1:41317
2016-05-05 22:53:54,884 INFO [utils.nio.NioClient] (NioClientHandler-3:) Connecting to 127.0.0.1:41317
2016-05-05 22:53:54,878 INFO [utils.nio.NioClient] (NioClientHandler-2:) Connecting to 127.0.0.1:41317
2016-05-05 22:53:54,877 INFO [utils.nio.NioClient] (NioClientHandler-1:) Connecting to 127.0.0.1:41317
2016-05-05 22:53:54,899 DEBUG [utils.crypt.EncryptionSecretKeyChecker] (pool-1-thread-1:) Encryption Type: null
2016-05-05 22:53:54,902 WARN [utils.nio.Link] (pool-1-thread-1:) SSL: Fail to find the generated keystore. Loading fail-safe one to continue.
2016-05-05 22:53:55,039 WARN [utils.nio.Link] (pool-1-thread-1:) SSL: Fail to find the generated keystore. Loading fail-safe one to continue.
2016-05-05 22:53:55,045 WARN [utils.nio.Link] (pool-1-thread-1:) SSL: Fail to find the generated keystore. Loading fail-safe one to continue.
2016-05-05 22:53:55,054 WARN [utils.nio.Link] (pool-1-thread-1:) SSL: Fail to find the generated keystore. Loading fail-safe one to continue.
2016-05-05 22:53:55,112 WARN [utils.nio.Link] (pool-1-thread-1:) SSL: Fail to find the generated keystore. Loading fail-safe one to continue.
2016-05-05 22:53:55,119 WARN [utils.nio.Link] (pool-1-thread-1:) SSL: Fail to find the generated keystore. Loading fail-safe one to continue.
2016-05-05 22:53:55,126 WARN [utils.nio.Link] (pool-1-thread-1:) SSL: Fail to find the generated keystore. Loading fail-safe one to continue.
2016-05-05 22:53:55,145 WARN [utils.nio.Link] (pool-1-thread-1:) SSL: Fail to find the generated keystore. Loading fail-safe one to continue.
2016-05-05 22:53:55,886 DEBUG [utils.testcase.NioTest] (Time-limited test:) 0/4 tests done. Waiting for completion
2016-05-05 22:53:56,152 INFO [utils.nio.NioClient] (NioClientHandler-3:) SSL: Handshake done
2016-05-05 22:53:56,152 INFO [utils.nio.NioClient] (NioClientHandler-3:) Connected to 127.0.0.1:41317
2016-05-05 22:53:56,198 INFO [utils.testcase.NioTest] (NioTestClient-2-Handler-1:) Client: Received CONNECT task
2016-05-05 22:53:56,258 INFO [utils.testcase.NioTest] (NioTestClient-2-Handler-1:) Sending data to server
2016-05-05 22:53:56,236 INFO [utils.nio.NioClient] (NioClientHandler-1:) SSL: Handshake done
2016-05-05 22:53:56,259 INFO [utils.nio.NioClient] (NioClientHandler-1:) Connected to 127.0.0.1:41317
2016-05-05 22:53:56,232 INFO [utils.nio.NioClient] (NioClientHandler-4:) SSL: Handshake done
2016-05-05 22:53:56,260 INFO [utils.nio.NioClient] (NioClientHandler-4:) Connected to 127.0.0.1:41317
2016-05-05 22:53:56,225 INFO [utils.nio.NioClient] (NioClientHandler-2:) SSL: Handshake done
2016-05-05 22:53:56,260 INFO [utils.nio.NioClient] (NioClientHandler-2:) Connected to 127.0.0.1:41317
2016-05-05 22:53:56,260 INFO [utils.testcase.NioTest] (NioTestClient-0-Handler-1:) Client: Received CONNECT task
2016-05-05 22:53:56,285 INFO [utils.testcase.NioTest] (NioTestClient-0-Handler-1:) Sending data to server
2016-05-05 22:53:56,285 INFO [utils.testcase.NioTest] (NioTestClient-3-Handler-1:) Client: Received CONNECT task
2016-05-05 22:53:56,331 INFO [utils.testcase.NioTest] (NioTestClient-3-Handler-1:) Sending data to server
2016-05-05 22:53:56,284 INFO [utils.testcase.NioTest] (NioTestClient-1-Handler-1:) Client: Received CONNECT task
2016-05-05 22:53:56,368 INFO [utils.testcase.NioTest] (NioTestClient-1-Handler-1:) Sending data to server
2016-05-05 22:53:56,286 INFO [utils.testcase.NioTest] (NioTestServer-Handler-1:) Server: Received CONNECT task
2016-05-05 22:53:56,487 INFO [utils.testcase.NioTest] (NioTestServer-Handler-1:) Server: Received CONNECT task
2016-05-05 22:53:56,487 INFO [utils.testcase.NioTest] (NioTestServer-Handler-1:) Server: Received CONNECT task
2016-05-05 22:53:56,487 INFO [utils.testcase.NioTest] (NioTestServer-Handler-1:) Server: Received CONNECT task
2016-05-05 22:53:56,999 DEBUG [utils.testcase.NioTest] (Time-limited test:) 0/4 tests done. Waiting for completion
2016-05-05 22:53:57,319 INFO [utils.testcase.NioTest] (NioTestServer-Handler-1:) Server: Received DATA task
2016-05-05 22:53:57,331 INFO [utils.testcase.NioTest] (NioTestServer-Handler-1:) Verify data received by server done.
2016-05-05 22:53:57,331 INFO [utils.testcase.NioTest] (NioTestServer-Handler-1:) Server: Received DATA task
2016-05-05 22:53:57,332 INFO [utils.testcase.NioTest] (NioTestServer-Handler-1:) Verify data received by server done.
2016-05-05 22:53:57,332 INFO [utils.testcase.NioTest] (NioTestServer-Handler-1:) Server: Received DATA task
2016-05-05 22:53:57,333 INFO [utils.testcase.NioTest] (NioTestServer-Handler-1:) Verify data received by server done.
2016-05-05 22:53:57,333 INFO [utils.testcase.NioTest] (NioTestServer-Handler-1:) Server: Received DATA task
2016-05-05 22:53:57,334 INFO [utils.testcase.NioTest] (NioTestServer-Handler-1:) Verify data received by server done.
2016-05-05 22:53:58,003 DEBUG [utils.testcase.NioTest] (Time-limited test:) 4/4 tests done.
2016-05-05 22:53:58,003 INFO [utils.testcase.NioTest] (main:) Clients stopped.
2016-05-05 22:53:58,004 INFO [utils.testcase.NioTest] (main:) Server stopped.
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.18 sec - in com.cloud.utils.testcase.NioTest
```
I have CI running against it now, so I will have results for the full CI run soon...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9348: Optimize NioTest and Nio...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/cloudstack/pull/1534
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9348: Optimize NioTest and Nio...
Posted by swill <gi...@git.apache.org>.
Github user swill commented on the pull request:
https://github.com/apache/cloudstack/pull/1534#issuecomment-217519767
### CI RESULTS
```
Tests Run: 85
Skipped: 0
Failed: 2
Errors: 0
Duration: 4h 31m 04s
```
**Summary of the problem(s):**
```
FAIL: Test redundant router internals
----------------------------------------------------------------------
Traceback (most recent call last):
File "/data/git/cs1/cloudstack/test/integration/smoke/test_routers_network_ops.py", line 290, in test_01_RVR_Network_FW_PF_SSH_default_routes_egress_true
"Attempt to retrieve google.com index page should be successful!"
AssertionError: Attempt to retrieve google.com index page should be successful!
----------------------------------------------------------------------
Additional details in: /tmp/MarvinLogs/test_network_C0JJZR/results.txt
```
```
FAIL: test_02_vpc_privategw_static_routes (integration.smoke.test_privategw_acl.TestPrivateGwACL)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/data/git/cs1/cloudstack/test/integration/smoke/test_privategw_acl.py", line 253, in test_02_vpc_privategw_static_routes
self.performVPCTests(vpc_off)
File "/data/git/cs1/cloudstack/test/integration/smoke/test_privategw_acl.py", line 304, in performVPCTests
privateGw_1 = self.createPvtGw(vpc_1, "10.0.3.100", "10.0.3.101", acl1.id, vlan_1)
File "/data/git/cs1/cloudstack/test/integration/smoke/test_privategw_acl.py", line 472, in createPvtGw
self.fail("Failed to create Private Gateway ==> %s" % e)
AssertionError: Failed to create Private Gateway ==> Execute cmd: createprivategateway failed, due to: errorCode: 431, errorText:Network with vlan vlan://100 already exists in zone 1
----------------------------------------------------------------------
Additional details in: /tmp/MarvinLogs/test_network_C0JJZR/results.txt
```
**Associated Uploads**
**`/tmp/MarvinLogs/DeployDataCenter__May_06_2016_15_29_07_FHJDSL:`**
* [dc_entries.obj](https://objects-east.cloud.ca/v1/e465abe2f9ae4478b9fff416eab61bd9/PR1534/tmp/MarvinLogs/DeployDataCenter__May_06_2016_15_29_07_FHJDSL/dc_entries.obj)
* [failed_plus_exceptions.txt](https://objects-east.cloud.ca/v1/e465abe2f9ae4478b9fff416eab61bd9/PR1534/tmp/MarvinLogs/DeployDataCenter__May_06_2016_15_29_07_FHJDSL/failed_plus_exceptions.txt)
* [runinfo.txt](https://objects-east.cloud.ca/v1/e465abe2f9ae4478b9fff416eab61bd9/PR1534/tmp/MarvinLogs/DeployDataCenter__May_06_2016_15_29_07_FHJDSL/runinfo.txt)
**`/tmp/MarvinLogs/test_network_C0JJZR:`**
* [failed_plus_exceptions.txt](https://objects-east.cloud.ca/v1/e465abe2f9ae4478b9fff416eab61bd9/PR1534/tmp/MarvinLogs/test_network_C0JJZR/failed_plus_exceptions.txt)
* [results.txt](https://objects-east.cloud.ca/v1/e465abe2f9ae4478b9fff416eab61bd9/PR1534/tmp/MarvinLogs/test_network_C0JJZR/results.txt)
* [runinfo.txt](https://objects-east.cloud.ca/v1/e465abe2f9ae4478b9fff416eab61bd9/PR1534/tmp/MarvinLogs/test_network_C0JJZR/runinfo.txt)
**`/tmp/MarvinLogs/test_vpc_routers_7C30C4:`**
* [failed_plus_exceptions.txt](https://objects-east.cloud.ca/v1/e465abe2f9ae4478b9fff416eab61bd9/PR1534/tmp/MarvinLogs/test_vpc_routers_7C30C4/failed_plus_exceptions.txt)
* [results.txt](https://objects-east.cloud.ca/v1/e465abe2f9ae4478b9fff416eab61bd9/PR1534/tmp/MarvinLogs/test_vpc_routers_7C30C4/results.txt)
* [runinfo.txt](https://objects-east.cloud.ca/v1/e465abe2f9ae4478b9fff416eab61bd9/PR1534/tmp/MarvinLogs/test_vpc_routers_7C30C4/runinfo.txt)
Uploads will be available until `2016-07-06 02:00:00 +0200 CEST`
*Comment created by [`upr comment`](https://github.com/cloudops/upr).*
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9348: Optimize NioTest and Nio...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on the pull request:
https://github.com/apache/cloudstack/pull/1534#issuecomment-217413262
@swill this PR is ready for CI test run and merge, thanks
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---