[GitHub] [beam] aromanenko-dev commented on a change in pull request #11090: [BEAM-9470] :sdks:java:io:kinesis:test is flaky

[GitBox <gi...@apache.org>]

aromanenko-dev commented on a change in pull request #11090: [BEAM-9470] :sdks:java:io:kinesis:test is flaky
URL: https://github.com/apache/beam/pull/11090#discussion_r391151296
 
 

 ##########
 File path: sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/ShardReadersPoolTest.java
 ##########
 @@ -324,10 +325,15 @@ public void shouldCallRateLimitPolicy()
       }
     }
 
+    ArgumentCaptor<List<KinesisRecord>> recordsCaptor = ArgumentCaptor.forClass(List.class);
+    verify(customRateLimitPolicy, atLeastOnce()).onSuccess(recordsCaptor.capture());
+    List<List<KinesisRecord>> capturedRecords = recordsCaptor.getAllValues();
+    assertThat(capturedRecords).contains(
+        ImmutableList.of(a, b),
+        singletonList(c),
+        singletonList(d),
+        Collections.emptyList()
+    );
     verify(customRateLimitPolicy).onThrottle(same(e));
-    verify(customRateLimitPolicy).onSuccess(eq(ImmutableList.of(a, b)));
-    verify(customRateLimitPolicy).onSuccess(eq(singletonList(c)));
-    verify(customRateLimitPolicy).onSuccess(eq(singletonList(d)));
 
 Review comment:
   I did run this test thousands of times locally (with `@RepeatedTest(n)` and some hacks to run it on Junit5) and I've managed to reproduce it only twice with the similar but a bit different error as it was on Jenkins.  
   
   Jenkins error: https://gist.github.com/aromanenko-dev/f9714c3dd5a47a9439b9245f3ef7b04b
   My local error (old test version): https://gist.github.com/aromanenko-dev/9a8acf265b96e010ad0ad1d09bb3ab92
   
   The common thing there is that it starts to emit empty records list after it returned a list with `c` value.
   
   Also, I did the same thing for new version of test and it's also reproducible (2 fails over 1000 runs):
   My local error (new test version) 1st fail: https://gist.github.com/aromanenko-dev/faa4229a486117d7504e486282503009
   My local error (new test version) 2nd fail: https://gist.github.com/aromanenko-dev/751cd007bb89a4f6e67f1ce9f9ef3d25
   
   As you can see the actually the errors with old and new versions tests are similar. So, it seems that it's not related to `verify` using and more related to how every mock `ShardRecordsIterator` returns the records. For some reasons it starts to return just empty list.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services