You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "KeDeng (Code Review)" <ge...@cloudera.org> on 2023/08/04 06:47:04 UTC
[kudu-CR] KUDU-3432: Fix flakiness in client-test.cc
KeDeng has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20314
Change subject: KUDU-3432: Fix flakiness in client-test.cc
......................................................................
KUDU-3432: Fix flakiness in client-test.cc
The 'TestGetTableKeyRange' in 'client-test.cc' may
be unstable in certain cases. Through analysis, it
was determined that the split algorithm might encounter
some corner cases when handling written data.
To mitigate the impact of these cases on CI, I have
increased the amount of data written in the unit tests
and modified some logic in dance to enhance the robustness
of the unit tests.
Change-Id: I1e3fecc09f58d91fe9a8c6d20caa21938dd9c972
---
M src/kudu/client/client-test.cc
1 file changed, 20 insertions(+), 27 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/14/20314/1
--
To view, visit http://gerrit.cloudera.org:8080/20314
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I1e3fecc09f58d91fe9a8c6d20caa21938dd9c972
Gerrit-Change-Number: 20314
Gerrit-PatchSet: 1
Gerrit-Owner: KeDeng <kd...@gmail.com>
[kudu-CR] KUDU-3432: Fix flakiness in client-test.cc
Posted by "KeDeng (Code Review)" <ge...@cloudera.org>.
Hello Alexey Serbin, Yingchun Lai, Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20314
to look at the new patch set (#2).
Change subject: KUDU-3432: Fix flakiness in client-test.cc
......................................................................
KUDU-3432: Fix flakiness in client-test.cc
The 'TestGetTableKeyRange' in 'client-test.cc' may
be unstable in certain cases. Through analysis, it
was determined that the split algorithm might encounter
some corner cases when handling written data.
To mitigate the impact of these cases on CI, I have
increased the amount of data written in the unit tests
and modified some logic in dance to enhance the robustness
of the unit tests.
Change-Id: I1e3fecc09f58d91fe9a8c6d20caa21938dd9c972
---
M src/kudu/client/client-test.cc
1 file changed, 22 insertions(+), 29 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/14/20314/2
--
To view, visit http://gerrit.cloudera.org:8080/20314
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I1e3fecc09f58d91fe9a8c6d20caa21938dd9c972
Gerrit-Change-Number: 20314
Gerrit-PatchSet: 2
Gerrit-Owner: KeDeng <kd...@gmail.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Yingchun Lai <la...@apache.org>
[kudu-CR] KUDU-3432: Fix flakiness in client-test.cc
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/20314 )
Change subject: KUDU-3432: Fix flakiness in client-test.cc
......................................................................
Patch Set 2: Code-Review+2
OK, this is possible to check this test scenario for stability even without using dist-test. I used the following
./bin/client-test --gtest_filter=TableKeyRangeTest.TestGetTableKeyRange --gtest_repeat=100 --stress_cpu_threads=8
Prior to PS2, I could see the following errors (maybe, one in 50 runs when running the bits compiled in RELEASE configuration):
src/kudu/client/client-test.cc:8788: Failure
Expected: (4) < (token_size_a), actual: 4 vs
With PS2, I ran it a few hundred times and didn't see a single failure.
Thank you for the fix!
--
To view, visit http://gerrit.cloudera.org:8080/20314
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1e3fecc09f58d91fe9a8c6d20caa21938dd9c972
Gerrit-Change-Number: 20314
Gerrit-PatchSet: 2
Gerrit-Owner: KeDeng <kd...@gmail.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: KeDeng <kd...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Yingchun Lai <la...@apache.org>
Gerrit-Comment-Date: Tue, 22 Aug 2023 01:31:23 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-3432: Fix flakiness in client-test.cc
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20314 )
Change subject: KUDU-3432: Fix flakiness in client-test.cc
......................................................................
KUDU-3432: Fix flakiness in client-test.cc
The 'TestGetTableKeyRange' in 'client-test.cc' may
be unstable in certain cases. Through analysis, it
was determined that the split algorithm might encounter
some corner cases when handling written data.
To mitigate the impact of these cases on CI, I have
increased the amount of data written in the unit tests
and modified some logic in dance to enhance the robustness
of the unit tests.
Change-Id: I1e3fecc09f58d91fe9a8c6d20caa21938dd9c972
Reviewed-on: http://gerrit.cloudera.org:8080/20314
Reviewed-by: Alexey Serbin <al...@apache.org>
Tested-by: Alexey Serbin <al...@apache.org>
---
M src/kudu/client/client-test.cc
1 file changed, 22 insertions(+), 29 deletions(-)
Approvals:
Alexey Serbin: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/20314
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I1e3fecc09f58d91fe9a8c6d20caa21938dd9c972
Gerrit-Change-Number: 20314
Gerrit-PatchSet: 3
Gerrit-Owner: KeDeng <kd...@gmail.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: KeDeng <kd...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Yingchun Lai <la...@apache.org>
[kudu-CR] KUDU-3432: Fix flakiness in client-test.cc
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/20314 )
Change subject: KUDU-3432: Fix flakiness in client-test.cc
......................................................................
Patch Set 2: Verified+1
unrelated test failure in TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData
--
To view, visit http://gerrit.cloudera.org:8080/20314
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1e3fecc09f58d91fe9a8c6d20caa21938dd9c972
Gerrit-Change-Number: 20314
Gerrit-PatchSet: 2
Gerrit-Owner: KeDeng <kd...@gmail.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: KeDeng <kd...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Yingchun Lai <la...@apache.org>
Gerrit-Comment-Date: Tue, 22 Aug 2023 01:31:55 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-3432: Fix flakiness in client-test.cc
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has removed a vote on this change.
Change subject: KUDU-3432: Fix flakiness in client-test.cc
......................................................................
Removed Verified-1 by Kudu Jenkins (120)
--
To view, visit http://gerrit.cloudera.org:8080/20314
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: I1e3fecc09f58d91fe9a8c6d20caa21938dd9c972
Gerrit-Change-Number: 20314
Gerrit-PatchSet: 2
Gerrit-Owner: KeDeng <kd...@gmail.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: KeDeng <kd...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Yingchun Lai <la...@apache.org>
[kudu-CR] KUDU-3432: Fix flakiness in client-test.cc
Posted by "KeDeng (Code Review)" <ge...@cloudera.org>.
KeDeng has posted comments on this change. ( http://gerrit.cloudera.org:8080/20314 )
Change subject: KUDU-3432: Fix flakiness in client-test.cc
......................................................................
Patch Set 2:
I tried the test command you provided in my environment, but it failed with the following error:
`
# KUDU_ALLOW_SLOW_TESTS=1 ../../build-support/dist_test.py loop -n 32 ./bin/client-test --gtest_filter=TableKeyRangeTest.TestGetTableKeyRange
2023-08-15 15:24:57,758 ERROR: Failed to run isolate
Traceback (most recent call last):
File "../../build-support/dist_test.py", line 749, in <module>
main(sys.argv[1:])
File "../../build-support/dist_test.py", line 745, in main
args.func(p, args)
File "../../build-support/dist_test.py", line 635, in loop_test
run_isolate(staging)
File "../../build-support/dist_test.py", line 494, in run_isolate
'--'] + staging.gen_json_paths())
File "/usr/lib64/python2.7/subprocess.py", line 537, in check_call
retcode = call(*popenargs, **kwargs)
File "/usr/lib64/python2.7/subprocess.py", line 524, in call
return Popen(*popenargs, **kwargs).wait()
File "/usr/lib64/python2.7/subprocess.py", line 711, in __init__
errread, errwrite)
File "/usr/lib64/python2.7/subprocess.py", line 1327, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
`
Thanks for your reviews.
It seems that my environment cannot access the address http://isolate.cloudera.org:4242/.
However, based on your feedback, I have made further modifications to the corner cases and conducted multiple tests using my locally developed script, and I did not encounter any more failures.
If it is convenient for you, could you please help verify the new submission again? I would greatly appreciate it.
--
To view, visit http://gerrit.cloudera.org:8080/20314
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1e3fecc09f58d91fe9a8c6d20caa21938dd9c972
Gerrit-Change-Number: 20314
Gerrit-PatchSet: 2
Gerrit-Owner: KeDeng <kd...@gmail.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: KeDeng <kd...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Yingchun Lai <la...@apache.org>
Gerrit-Comment-Date: Tue, 15 Aug 2023 08:08:52 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-3432: Fix flakiness in client-test.cc
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/20314 )
Change subject: KUDU-3432: Fix flakiness in client-test.cc
......................................................................
Patch Set 1:
Thank you for working on the fix.
It seems that even with this update the test is still flaky, failing time to time with the following error:
src/kudu/client/client-test.cc:8776: Failure Expected: (4) < (token_size_a), actual: 4 vs 4
To reproduce, I applied your patch, built client-test in RELEASE configuration, and ran it via dist-test using the following command:
KUDU_ALLOW_SLOW_TESTS=1 ../../build-support/dist_test.py loop -n 32 ./bin/client-test --gtest_filter=TableKeyRangeTest.TestGetTableKeyRange
The stats/results for that run can be seen at http://dist-test.cloudera.org/job?job_id=aserbin.1691173568.105292
--
To view, visit http://gerrit.cloudera.org:8080/20314
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1e3fecc09f58d91fe9a8c6d20caa21938dd9c972
Gerrit-Change-Number: 20314
Gerrit-PatchSet: 1
Gerrit-Owner: KeDeng <kd...@gmail.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Yingchun Lai <la...@apache.org>
Gerrit-Comment-Date: Fri, 04 Aug 2023 18:33:26 +0000
Gerrit-HasComments: No