You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2020/04/26 01:29:18 UTC
[GitHub] [kafka] jiameixie opened a new pull request #8555: KAFKA-9920:Fix NetworkDegradeTest.test_rate test error
jiameixie opened a new pull request #8555:
URL: https://github.com/apache/kafka/pull/8555
The test case of kafkatest.tests.core.network_degrade_test.NetworkDegradeTest.test_rate.
rate_limit_kbit=1000000.device_name=eth0.task_name=rate-1000-latency-50.latency_ms=50
failed. And the error log was "Expected most of the measured rates to be within an order
of magnitude of target 1000000. This means `tc` did not limit the bandwidth as expected."
It was because that the rate_limt didn't take immediately after starting.
Change-Id: I2de1d3fc696e11b411986bb2e3ce43f074fbac4a
Signed-off-by: Jiamei Xie <ji...@arm.com>
*More detailed description of your change,
if necessary. The PR title and PR message become
the squashed commit message, so use a separate
comment to ping reviewers.*
*Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.*
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mjsax commented on pull request #8555: KAFKA-9920:Fix NetworkDegradeTest.test_rate test error
Posted by GitBox <gi...@apache.org>.
mjsax commented on pull request #8555:
URL: https://github.com/apache/kafka/pull/8555#issuecomment-622648385
I am not familiar with this code. Maybe @mumrah or @cmccabe can help?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jiameixie commented on pull request #8555: KAFKA-9920:Fix NetworkDegradeTest.test_rate test error
Posted by GitBox <gi...@apache.org>.
jiameixie commented on pull request #8555:
URL: https://github.com/apache/kafka/pull/8555#issuecomment-619464716
The error log is:
network_degrade_test - test_rate - lineno:130]: Measured rates: [953747.0, 21894.0, 33883.0, 32319.0, 33362.0, 33362.0, 33362.0, 33362.0, 32841.0, 33362.0, 33883.0, 32319.0, 33883.0, 33362.0, 33362.0, 33362.0, 32841.0, 33883.0, 15638.0, 33883.0, 77799.0]
[INFO - 2020-04-23 19:22:17,401 - runner_client - log - lineno:237]: RunnerClient: kafkatest.tests.core.network_degrade_test.NetworkDegradeTest.test_rate.rate_limit_kbit=1000000.device_name=eth0.task_name=rate-1000-latency-50.latency_ms=50: FAIL: Expected most of the measured rates to be within an order of magnitude of target 1000000. This means `tc` did not limit the bandwidth as expected.
And I looked into the code. Following are its simple workflow(zookeepers has been started before it)
#ducker02 and ducker03 are containers
1. on ducker02 (delay tc)
sudo tc qdisc add dev eth0 root handle 1:0 netem delay 50ms 7ms distribution paretonormal
2. on ducker02 (rate tc)
sudo tc qdisc add dev eth0 parent 1:1 handle 10: tbf rate 1000000kbit burst 1mbit latency 500ms
3. on ducker03 (iperf server)
sudo iperf -s
4. on duker02 (iperf client)
iperf -i 1 -t 20 -f k -c ducker03
5. Collect iperf results and check the measured_rates. Below is its logic to decide whether the test pass.
# We expect to see measured rates within an order of magnitude of our target rate
low_kbps = rate_limit_kbit / 10
high_kbps = rate_limit_kbit * 10
acceptable_rates = [r for r in measured_rates if low_kbps < r < high_kbps]
assert len(acceptable_rates) > 5, msg
#run iperf without rate or latency limit
ducker@ducker02:/$ iperf -i 1 -t 20 -f k -c ducker03
------------------------------------------------------------
Client connecting to ducker03, TCP port 5001
TCP window size: 9766 KByte (default)
------------------------------------------------------------
[ 3] local 172.30.0.3 port 60414 connected with 172.30.0.4 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 1.0 sec 2444032 KBytes 20021510 Kbits/sec
[ 3] 1.0- 2.0 sec 3021312 KBytes 24750588 Kbits/sec
[ 3] 2.0- 3.0 sec 3131904 KBytes 25656558 Kbits/sec
[ 3] 3.0- 4.0 sec 3626496 KBytes 29708255 Kbits/sec
[ 3] 4.0- 5.0 sec 3556608 KBytes 29135733 Kbits/sec
[ 3] 5.0- 6.0 sec 3592960 KBytes 29433528 Kbits/sec
[ 3] 6.0- 7.0 sec 3586048 KBytes 29376905 Kbits/sec
[ 3] 7.0- 8.0 sec 3351168 KBytes 27452768 Kbits/sec
^C[ 3] 0.0- 8.1 sec 26616960 KBytes 26978713 Kbits/sec
#add limit and run iperf imediately.
ducker@ducker02:/$ sudo tc qdisc add dev eth0 root handle 1:0 netem delay 50ms 7ms distribution paretonormal
ducker@ducker02:/$ sudo tc qdisc add dev eth0 parent 1:1 handle 10: tbf rate 1000000kbit burst 1mbit latency 500ms
ducker@ducker02:/$ iperf -i 1 -t 20 -f k -c ducker03
------------------------------------------------------------
Client connecting to ducker03, TCP port 5001
TCP window size: 9766 KByte (default)
------------------------------------------------------------
[ 3] local 172.30.0.3 port 60432 connected with 172.30.0.4 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 1.0 sec 10969 KBytes 89857 Kbits/sec
[ 3] 1.0- 2.0 sec 5854 KBytes 47958 Kbits/sec
[ 3] 2.0- 3.0 sec 6109 KBytes 50043 Kbits/sec
[ 3] 3.0- 4.0 sec 6382 KBytes 52278 Kbits/sec
[ 3] 4.0- 5.0 sec 6409 KBytes 52499 Kbits/sec
[ 3] 5.0- 6.0 sec 6656 KBytes 54526 Kbits/sec
[ 3] 6.0- 7.0 sec 9728 KBytes 79692 Kbits/sec
[ 3] 7.0- 8.0 sec 6656 KBytes 54526 Kbits/sec
[ 3] 8.0- 9.0 sec 6656 KBytes 54526 Kbits/sec
[ 3] 9.0-10.0 sec 9856 KBytes 80740 Kbits/sec
[ 3] 10.0-11.0 sec 9728 KBytes 79692 Kbits/sec
[ 3] 11.0-12.0 sec 9856 KBytes 80740 Kbits/sec
[ 3] 12.0-13.0 sec 9856 KBytes 80740 Kbits/sec
[ 3] 13.0-14.0 sec 9728 KBytes 79692 Kbits/sec
[ 3] 14.0-15.0 sec 9856 KBytes 80740 Kbits/sec
[ 3] 15.0-16.0 sec 10112 KBytes 82838 Kbits/sec
[ 3] 16.0-17.0 sec 13184 KBytes 108003 Kbits/sec
[ 3] 17.0-18.0 sec 13184 KBytes 108003 Kbits/sec
[ 3] 18.0-19.0 sec 16512 KBytes 135266 Kbits/sec
[ 3] 19.0-20.0 sec 16640 KBytes 136315 Kbits/sec
[ 3] 0.0-20.2 sec 193930 KBytes 78783 Kbits/sec
# wait for a while, run iperf again
ducker@ducker02:/$ iperf -i 1 -t 20 -f k -c ducker03
------------------------------------------------------------
Client connecting to ducker03, TCP port 5001
TCP window size: 9766 KByte (default)
------------------------------------------------------------
[ 3] local 172.30.0.3 port 60452 connected with 172.30.0.4 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 1.0 sec 15553 KBytes 127412 Kbits/sec
[ 3] 1.0- 2.0 sec 13440 KBytes 110100 Kbits/sec
[ 3] 2.0- 3.0 sec 13056 KBytes 106955 Kbits/sec
[ 3] 3.0- 4.0 sec 13696 KBytes 112198 Kbits/sec
[ 3] 4.0- 5.0 sec 12928 KBytes 105906 Kbits/sec
[ 3] 5.0- 6.0 sec 10112 KBytes 82838 Kbits/sec
[ 3] 6.0- 7.0 sec 16512 KBytes 135266 Kbits/sec
[ 3] 7.0- 8.0 sec 13440 KBytes 110100 Kbits/sec
[ 3] 8.0- 9.0 sec 13184 KBytes 108003 Kbits/sec
[ 3] 9.0-10.0 sec 13056 KBytes 106955 Kbits/sec
[ 3] 10.0-11.0 sec 16640 KBytes 136315 Kbits/sec
[ 3] 11.0-12.0 sec 13056 KBytes 106955 Kbits/sec
[ 3] 12.0-13.0 sec 13568 KBytes 111149 Kbits/sec
[ 3] 13.0-14.0 sec 16640 KBytes 136315 Kbits/sec
[ 3] 14.0-15.0 sec 16640 KBytes 136315 Kbits/sec
[ 3] 15.0-16.0 sec 13184 KBytes 108003 Kbits/sec
[ 3] 16.0-17.0 sec 16512 KBytes 135266 Kbits/sec
[ 3] 17.0-18.0 sec 16896 KBytes 138412 Kbits/sec
[ 3] 18.0-19.0 sec 16384 KBytes 134218 Kbits/sec
[ 3] 19.0-20.0 sec 16256 KBytes 133169 Kbits/sec
[ 3] 0.0-20.1 sec 290753 KBytes 118465 Kbits/sec
The rate limit is 1000000kbit. So the acceptable_rates list include rates between 100000 and 10000000. And test pass if the acceptable_rates len is bigger than 5.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jiameixie commented on pull request #8555: KAFKA-9920:Fix NetworkDegradeTest.test_rate test error
Posted by GitBox <gi...@apache.org>.
jiameixie commented on pull request #8555:
URL: https://github.com/apache/kafka/pull/8555#issuecomment-619464996
@abbccdda @mjsax PTAL, thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on pull request #8555: KAFKA-9920:Fix NetworkDegradeTest.test_rate test error
Posted by GitBox <gi...@apache.org>.
cmccabe commented on pull request #8555:
URL: https://github.com/apache/kafka/pull/8555#issuecomment-737380941
I'm not familiar with this either. How did you choose a 15 second delay?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org