[kudu-CR] [tool-test] add scenario for KUDU-2819 regressions

["Alexey Serbin (Code Review)" <ge...@cloudera.org>]

Hello Will Berkeley, Adar Dembo, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/13380

to look at the new patch set (#3).

Change subject: [tool-test] add scenario for KUDU-2819 regressions
......................................................................

[tool-test] add scenario for KUDU-2819 regressions

Added a test scenario to check how ksck works in case of
collecting information from many tablets servers a Kudu cluster.

This is to cover regressions for a race condition that existed in
FetchInfoFromTabletServers() some time ago before it was fixed with
d17f9fce6 (along with bringing other improvements).  One manifestation
of the race condition is described in KUDU-2819.

With this test, I verified that the version prior to the above mentioned
patch d17f9fce6 was susceptible to the race as reported by sanitizers:

  ThreadSanitizer (100% failure rate):
    http://dist-test.cloudera.org/job?job_id=aserbin.1558485044.66856

  AddressSanitizer (~1 out of 20 failed), reproducing double-free,
  heap-buffer-overflow, memory leaks and friends:
    http://dist-test.cloudera.org/job?job_id=aserbin.1558484756.62807

Also, I verified that by commenting out the guard that protects
concurrent calls of std::vector::emplace_back() on 'warning_messages'
container in FetchInfoFromTabletServers() method
(src/kudu/toolts/ksck.cc), both sanitizers were able to detect the race:

  ThreadSanitizer (100% failure rate):
    http://dist-test.cloudera.org/job?job_id=aserbin.1558485274.73242

  AddressSanitizer (~1 out of 20 failed):
    http://dist-test.cloudera.org/job?job_id=aserbin.1558485217.70253

Change-Id: I16cf69b6f7d2fb59014df26601dfc30e124a52ee
---
M src/kudu/tools/kudu-tool-test.cc
1 file changed, 41 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/80/13380/3
-- 
To view, visit http://gerrit.cloudera.org:8080/13380
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I16cf69b6f7d2fb59014df26601dfc30e124a52ee
Gerrit-Change-Number: 13380
Gerrit-PatchSet: 3
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>