You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Todd Lipcon (Code Review)" <ge...@cloudera.org> on 2017/07/25 06:13:27 UTC
[kudu-CR] KUDU-2053. Fix race in Java RequestTracker
Hello David Ribeiro Alves, Jean-Daniel Cryans,
I'd like you to do a code review. Please visit
http://gerrit.cloudera.org:8080/7494
to review the following change.
Change subject: KUDU-2053. Fix race in Java RequestTracker
......................................................................
KUDU-2053. Fix race in Java RequestTracker
The implementation of RequestTracker.newSeqNo() was previously
implemented as:
1. allocate the new sequence number
2. add that sequence number to the 'incomplete' map
These steps were individually thread-safe by using an AtomicLong and a
thread-safe collection, respectively, but they were not performed
atomically. Thus we could have the following race:
T1: allocate seq number 1
T2: allocate seq number 2
T2: add seq number 2 to incompleteRpcs
T2: ask for firstIncomplete() -> 2
T2: send an RPC
--> server GCs seqnum < 2
T1: add seq number 1 to incompleteRpcs
T1: send an RPC with seq number 1
--> server responds with an error since this seqnum is already
GCed
This patch fixes the issue by moving back to a simpler synchronization
scheme such that the two steps (allocation and addition to the tracking
structure) are done under a single critical section.
A new unit test is included which reliably reproduced the issue prior to
the fix.
Change-Id: I56f3d1ac85d34ca663e5b6378ff8362846a2424a
---
M java/kudu-client/src/main/java/org/apache/kudu/client/RequestTracker.java
M java/kudu-client/src/test/java/org/apache/kudu/client/TestRequestTracker.java
2 files changed, 70 insertions(+), 11 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/94/7494/1
--
To view, visit http://gerrit.cloudera.org:8080/7494
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I56f3d1ac85d34ca663e5b6378ff8362846a2424a
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: David Ribeiro Alves <da...@gmail.com>
Gerrit-Reviewer: Jean-Daniel Cryans <jd...@apache.org>
[kudu-CR] KUDU-2053. Fix race in Java RequestTracker
Posted by "Jean-Daniel Cryans (Code Review)" <ge...@cloudera.org>.
Jean-Daniel Cryans has posted comments on this change.
Change subject: KUDU-2053. Fix race in Java RequestTracker
......................................................................
Patch Set 1: Code-Review+2
(1 comment)
http://gerrit.cloudera.org:8080/#/c/7494/1/java/kudu-client/src/main/java/org/apache/kudu/client/RequestTracker.java
File java/kudu-client/src/main/java/org/apache/kudu/client/RequestTracker.java:
Line 67: synchronized (lock) {
Worth considering a RWLock?
--
To view, visit http://gerrit.cloudera.org:8080/7494
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I56f3d1ac85d34ca663e5b6378ff8362846a2424a
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: David Ribeiro Alves <da...@gmail.com>
Gerrit-Reviewer: Jean-Daniel Cryans <jd...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-HasComments: Yes
[kudu-CR] KUDU-2053. Fix race in Java RequestTracker
Posted by "Jean-Daniel Cryans (Code Review)" <ge...@cloudera.org>.
Jean-Daniel Cryans has posted comments on this change.
Change subject: KUDU-2053. Fix race in Java RequestTracker
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/7494/1/java/kudu-client/src/main/java/org/apache/kudu/client/RequestTracker.java
File java/kudu-client/src/main/java/org/apache/kudu/client/RequestTracker.java:
Line 67: synchronized (lock) {
> I think the ratio of firstIncomplete (read-only) to newSeqNo/rpcCompleted (
Sounds good, thanks for the explanation Todd and for fixing my horrible mistake :)
--
To view, visit http://gerrit.cloudera.org:8080/7494
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I56f3d1ac85d34ca663e5b6378ff8362846a2424a
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: David Ribeiro Alves <da...@gmail.com>
Gerrit-Reviewer: Jean-Daniel Cryans <jd...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-HasComments: Yes
[kudu-CR] KUDU-2053. Fix race in Java RequestTracker
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has submitted this change and it was merged.
Change subject: KUDU-2053. Fix race in Java RequestTracker
......................................................................
KUDU-2053. Fix race in Java RequestTracker
The implementation of RequestTracker.newSeqNo() was previously
implemented as:
1. allocate the new sequence number
2. add that sequence number to the 'incomplete' map
These steps were individually thread-safe by using an AtomicLong and a
thread-safe collection, respectively, but they were not performed
atomically. Thus we could have the following race:
T1: allocate seq number 1
T2: allocate seq number 2
T2: add seq number 2 to incompleteRpcs
T2: ask for firstIncomplete() -> 2
T2: send an RPC
--> server GCs seqnum < 2
T1: add seq number 1 to incompleteRpcs
T1: send an RPC with seq number 1
--> server responds with an error since this seqnum is already
GCed
This patch fixes the issue by moving back to a simpler synchronization
scheme such that the two steps (allocation and addition to the tracking
structure) are done under a single critical section.
A new unit test is included which reliably reproduced the issue prior to
the fix.
Change-Id: I56f3d1ac85d34ca663e5b6378ff8362846a2424a
Reviewed-on: http://gerrit.cloudera.org:8080/7494
Tested-by: Kudu Jenkins
Reviewed-by: Jean-Daniel Cryans <jd...@apache.org>
---
M java/kudu-client/src/main/java/org/apache/kudu/client/RequestTracker.java
M java/kudu-client/src/test/java/org/apache/kudu/client/TestRequestTracker.java
2 files changed, 70 insertions(+), 11 deletions(-)
Approvals:
Jean-Daniel Cryans: Looks good to me, approved
Kudu Jenkins: Verified
--
To view, visit http://gerrit.cloudera.org:8080/7494
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I56f3d1ac85d34ca663e5b6378ff8362846a2424a
Gerrit-PatchSet: 2
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: David Ribeiro Alves <da...@gmail.com>
Gerrit-Reviewer: Jean-Daniel Cryans <jd...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
[kudu-CR] KUDU-2053. Fix race in Java RequestTracker
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change.
Change subject: KUDU-2053. Fix race in Java RequestTracker
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/7494/1/java/kudu-client/src/main/java/org/apache/kudu/client/RequestTracker.java
File java/kudu-client/src/main/java/org/apache/kudu/client/RequestTracker.java:
Line 67: synchronized (lock) {
> Worth considering a RWLock?
I think the ratio of firstIncomplete (read-only) to newSeqNo/rpcCompleted (write) is actually <1.0, so don't think there is much benefit to be gained. One potential improvement if we see any contention here would be to have a single method like 'void initHeader(RequestHeaderPB pb)' which inits both the seq num and the firstIncomplete under a single lock acquisition (instead of two) but I'm going to hand-wave here and say that if it's beneficial the JVM is already equipped to do lock coarsening here (https://shipilev.net/jvm-anatomy-park/1-lock-coarsening-for-loops/)
--
To view, visit http://gerrit.cloudera.org:8080/7494
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I56f3d1ac85d34ca663e5b6378ff8362846a2424a
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: David Ribeiro Alves <da...@gmail.com>
Gerrit-Reviewer: Jean-Daniel Cryans <jd...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-HasComments: Yes