You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Todd Lipcon (Code Review)" <ge...@cloudera.org> on 2017/07/25 16:49:00 UTC

[kudu-CR](branch-1.3.x) KUDU-2053. Fix race in Java RequestTracker 

Todd Lipcon has uploaded a new change for review.

  http://gerrit.cloudera.org:8080/7497

Change subject: KUDU-2053. Fix race in Java RequestTracker
......................................................................

KUDU-2053. Fix race in Java RequestTracker

The implementation of RequestTracker.newSeqNo() was previously
implemented as:

1. allocate the new sequence number
2. add that sequence number to the 'incomplete' map

These steps were individually thread-safe by using an AtomicLong and a
thread-safe collection, respectively, but they were not performed
atomically. Thus we could have the following race:

T1: allocate seq number 1
T2:     allocate seq number 2
T2:     add seq number 2 to incompleteRpcs
T2:     ask for firstIncomplete() -> 2
T2:     send an RPC
        --> server GCs seqnum < 2
T1: add seq number 1 to incompleteRpcs
T1: send an RPC with seq number 1
    --> server responds with an error since this seqnum is already
        GCed

This patch fixes the issue by moving back to a simpler synchronization
scheme such that the two steps (allocation and addition to the tracking
structure) are done under a single critical section.

A new unit test is included which reliably reproduced the issue prior to
the fix.

Change-Id: I56f3d1ac85d34ca663e5b6378ff8362846a2424a
Reviewed-on: http://gerrit.cloudera.org:8080/7494
Tested-by: Kudu Jenkins
Reviewed-by: Jean-Daniel Cryans <jd...@apache.org>
(cherry picked from commit be8e3c22b9a3a71b2c365e2b9ed306ea23d60058)
---
M java/kudu-client/src/main/java/org/apache/kudu/client/RequestTracker.java
M java/kudu-client/src/test/java/org/apache/kudu/client/TestRequestTracker.java
2 files changed, 70 insertions(+), 11 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/97/7497/1
-- 
To view, visit http://gerrit.cloudera.org:8080/7497
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I56f3d1ac85d34ca663e5b6378ff8362846a2424a
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: branch-1.3.x
Gerrit-Owner: Todd Lipcon <to...@apache.org>

[kudu-CR](branch-1.3.x) KUDU-2053. Fix race in Java RequestTracker

Posted by "Jean-Daniel Cryans (Code Review)" <ge...@cloudera.org>.
Jean-Daniel Cryans has posted comments on this change.

Change subject: KUDU-2053. Fix race in Java RequestTracker
......................................................................


Patch Set 1: Code-Review+2

-- 
To view, visit http://gerrit.cloudera.org:8080/7497
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I56f3d1ac85d34ca663e5b6378ff8362846a2424a
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: branch-1.3.x
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Jean-Daniel Cryans <jd...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-HasComments: No

[kudu-CR](branch-1.3.x) KUDU-2053. Fix race in Java RequestTracker

Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has submitted this change and it was merged.

Change subject: KUDU-2053. Fix race in Java RequestTracker
......................................................................


KUDU-2053. Fix race in Java RequestTracker

The implementation of RequestTracker.newSeqNo() was previously
implemented as:

1. allocate the new sequence number
2. add that sequence number to the 'incomplete' map

These steps were individually thread-safe by using an AtomicLong and a
thread-safe collection, respectively, but they were not performed
atomically. Thus we could have the following race:

T1: allocate seq number 1
T2:     allocate seq number 2
T2:     add seq number 2 to incompleteRpcs
T2:     ask for firstIncomplete() -> 2
T2:     send an RPC
        --> server GCs seqnum < 2
T1: add seq number 1 to incompleteRpcs
T1: send an RPC with seq number 1
    --> server responds with an error since this seqnum is already
        GCed

This patch fixes the issue by moving back to a simpler synchronization
scheme such that the two steps (allocation and addition to the tracking
structure) are done under a single critical section.

A new unit test is included which reliably reproduced the issue prior to
the fix.

Change-Id: I56f3d1ac85d34ca663e5b6378ff8362846a2424a
Reviewed-on: http://gerrit.cloudera.org:8080/7494
Tested-by: Kudu Jenkins
Reviewed-by: Jean-Daniel Cryans <jd...@apache.org>
(cherry picked from commit be8e3c22b9a3a71b2c365e2b9ed306ea23d60058)
Reviewed-on: http://gerrit.cloudera.org:8080/7497
---
M java/kudu-client/src/main/java/org/apache/kudu/client/RequestTracker.java
M java/kudu-client/src/test/java/org/apache/kudu/client/TestRequestTracker.java
2 files changed, 70 insertions(+), 11 deletions(-)

Approvals:
  Jean-Daniel Cryans: Looks good to me, approved
  Kudu Jenkins: Verified



-- 
To view, visit http://gerrit.cloudera.org:8080/7497
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I56f3d1ac85d34ca663e5b6378ff8362846a2424a
Gerrit-PatchSet: 2
Gerrit-Project: kudu
Gerrit-Branch: branch-1.3.x
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Jean-Daniel Cryans <jd...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>