You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kudu.apache.org by "Todd Lipcon (Code Review)" <ge...@ip-10-146-233-104.ec2.internal> on 2016/01/15 03:25:45 UTC

[kudu-CR] Fix some TSAN failures seen on GCE

Hello Dan Burkert, Mike Percy, Adar Dembo,

I'd like you to do a code review.  Please visit

    http://gerrit.cloudera.org:8080/1793

to review the following change.

Change subject: Fix some TSAN failures seen on GCE
......................................................................

Fix some TSAN failures seen on GCE

This fixes a few TSAN issues that I've seen while testing on GCE:

- Add suppression for epoll_ctl

libev is missing TSAN annotations and thus its synchronization doesn't
get registered as "happens-before" relationships. This can cause spurious
TSAN errors relating to epoll file descriptors. Before this suppression,
I saw 8/100 runs of DeleteTableTest TSAN fail with such a race.

- Make churny election test a little less churny on TSAN

The TestChurnyElections test sets timeouts to 1ms. On TSAN on GCE instances
this is often failing because it doesn't make any progress at all. This
patch sets the election timeout to 5ms instead of 1ms and also disables
fsync, which is a bit slower on GCE due to storage being remote.

- Increase timeout waiting for WALs in some integration tests

This fixes some flakiness in delete_table-test TSAN builds where writes
were going slowly enough that we didn't reach the desired number of WALs
within 30 seconds. After doubling to 60 seconds, I didn't see failures.

- Bump test workload write timeout to 50ms in memory limit test

With a 10ms timeout, the client was often unable to even connect to the
server to start any transactions in TSAN builds. Bumping to 50ms makes
the test much more reliable.

- Add TSAN annotations around iterator stats

We assume that iterator stats can be fuzzy and don't want to add barriers
on the write side. This patch adds an IGNORE annotation around reading
the stats.

Change-Id: Idb7447b9b88c37a45be3840d5b24edafc237ad51
---
M build-support/tsan-suppressions.txt
M src/kudu/integration-tests/external_mini_cluster_fs_inspector.h
M src/kudu/integration-tests/raft_consensus-itest.cc
M src/kudu/tablet/cfile_set.cc
4 files changed, 13 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/93/1793/1
-- 
To view, visit http://gerrit.cloudera.org:8080/1793
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Idb7447b9b88c37a45be3840d5b24edafc237ad51
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@cloudera.com>
Gerrit-Reviewer: Mike Percy <mp...@cloudera.com>