[GitHub] [kafka] vvcephei commented on a change in pull request #11624: MINOR: add PAPI KV store tests for IQv2

[GitBox <gi...@apache.org>]

vvcephei commented on a change in pull request #11624:
URL: https://github.com/apache/kafka/pull/11624#discussion_r774097786



##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/IQv2StoreIntegrationTest.java
##########
@@ -287,31 +306,42 @@ public boolean keyValue() {
         }
     }
 
-    @Parameterized.Parameters(name = "cache={0}, log={1}, supplier={2}")
+    @Parameterized.Parameters(name = "cache={0}, log={1}, supplier={2}, kind={3}")
     public static Collection<Object[]> data() {
+        LOG.info("Generating test cases according to random seed: {}", SEED);
         final List<Object[]> values = new ArrayList<>();
         for (final boolean cacheEnabled : Arrays.asList(true, false)) {
             for (final boolean logEnabled : Arrays.asList(true, false)) {
                 for (final StoresToTest toTest : StoresToTest.values()) {
-                    values.add(new Object[]{cacheEnabled, logEnabled, toTest.name()});
+                    for (final String kind : Arrays.asList("DSL", "PAPI")) {
+                        values.add(new Object[]{cacheEnabled, logEnabled, toTest.name(), kind});
+                    }
                 }
             }
         }
+        // Randomizing the test cases in case some orderings interfere with each other.
+        // If you wish to reproduce a randomized order, copy the logged SEED and substitute
+        // it for the constant at the top of the file. This will cause exactly the same sequence
+        // of pseudorandom values to be generated.
+        Collections.shuffle(values, RANDOM);

Review comment:
       I added this because it occurred to me that there's a small chance that the tests only pass in the order we generate them in. By shuffling the cases, we ensure that every possible ordering will eventually be run. The price we pay is that if there's an ordering that causes the test to fail, it will appear to be flaky. For that reason, we initialize the Random with a seed and log it. As the comment states, you can re-run the exact same ordering by setting SEED to the one that was logged.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org