[jira] [Comment Edited] (OOZIE-3263) Flaky test TestZKUUIDService#testResetSequence_withMultiThread

["Peter Bacsko (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/OOZIE-3263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16492826#comment-16492826 ] 

Peter Bacsko edited comment on OOZIE-3263 at 5/28/18 4:52 PM:
--------------------------------------------------------------

I have a theory for this.

1. Thread#1 reaches end of the sequence (5000) and enters {{resetSequence()}}
2. Thread#2 also reaches the end of the sequence and enterd {{resetSequence()}} too
3. Thread#1 successfully acquires ZK lock and begins to perform resetting the distributed counter
4. Thread#2 tries to acquire the ZK lock in a loop but fails after multiple attempts.
5. An exception is thrown from {{resetSequence()}} which is caught by {{createSequence()}} and calls {{UUIDService.getCounter()}} as a fallback mechanism, which returns 0.
6. In the meantime, Thread #1 has returned 0 and the respective {{AtomicInteger}} has been incremented
7. Thread#2 also increments {{AtomicInteger}} at position 0 in the array

Step #4 seems to be highly unlikely. Lock timeout for ZK is 5 seconds and there are 3 attempts to acquire the lock. Even on a slow computer, it shouldn't take 15 seconds to complete {{resetSequence()}}, it sounds absurdly slow.

Regardless, we need more logs to find out if this sequence of events causes the test error.


was (Author: pbacsko):
I have a theory for this.

1. Thread#1 reaches end of the sequence (5000) and enters {{resetSequence()}}
2. Thread#2 also reaches the end of the sequence and enterd {{resetSequence()}} too
3. Thread#1 successfully acquires ZK lock and begins to perform resetting the distributed counter
4. Thread#2 tries to acquire the ZK lock in a loop but fails after multiple attempts.
5. An exception is thrown from {{resetSequence()}} which is caught by {{createSequence()}} and calls {{UUIDService.getCounter()}} as a fallback mechanism, which returns 0.
6. In the meantime, Thread #1 has returned 0 and the respective {{AtomicInteger}} has been incremented
7. Thread#2 also increments {{AtomicInteger}} at position 0 in the array

Step #4 seems to be highly unlikely. Lock timeout for ZK is 5 seconds and there are 3 attempts to acquire the lock. Even on a slow computer, it shouldn't take 15 seconds, it sounds unacceptably slow. 

Regardless, we need more logs to find out if this sequence of events causes the test error.

> Flaky test TestZKUUIDService#testResetSequence_withMultiThread
> --------------------------------------------------------------
>
>                 Key: OOZIE-3263
>                 URL: https://issues.apache.org/jira/browse/OOZIE-3263
>             Project: Oozie
>          Issue Type: Sub-task
>            Reporter: Peter Bacsko
>            Assignee: Peter Bacsko
>            Priority: Major
>
> The test case {{testResetSequence_withMultiThread}} failed with the following error:
> {noformat}
> junit.framework.AssertionFailedError: expected:<3> but was:<2>
> 	at junit.framework.Assert.fail(Assert.java:57)
> 	at junit.framework.Assert.failNotEquals(Assert.java:329)
> 	at junit.framework.Assert.assertEquals(Assert.java:78)
> 	at junit.framework.Assert.assertEquals(Assert.java:234)
> 	at junit.framework.Assert.assertEquals(Assert.java:241)
> 	at junit.framework.TestCase.assertEquals(TestCase.java:409)
> 	at org.apache.oozie.service.TestZKUUIDService.testResetSequence_withMultiThread(TestZKUUIDService.java:214)
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)