[GitHub] [kafka] kebab-mai-haddi opened a new pull request #10554: Expand AdjustStreamThreadCountTest by writing some data to Kafka topics

[GitBox <gi...@apache.org>]

kebab-mai-haddi opened a new pull request #10554:
URL: https://github.com/apache/kafka/pull/10554


   *More detailed description of your change,
   if necessary. The PR title and PR message become
   the squashed commit message, so use a separate
   comment to ping reviewers.*
   
   *Summary of testing strategy (including rationale)
   for the feature or bug fix. Unit and/or integration
   tests are expected for any behaviour change and
   system tests should be considered for larger changes.*
   
   ### Committer Checklist (excluded from commit message)
   - [ ] Verify design and implementation 
   - [ ] Verify test coverage and CI build status
   - [ ] Verify documentation (including upgrade notes)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [kafka] ableegoldman commented on a change in pull request #10554: Expand AdjustStreamThreadCountTest by writing some data to Kafka topics

[GitBox <gi...@apache.org>]

ableegoldman commented on a change in pull request #10554:
URL: https://github.com/apache/kafka/pull/10554#discussion_r620794381



##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/AdjustStreamThreadCountTest.java
##########
@@ -121,6 +125,21 @@ public void setup() {
         );
     }
 
+
+    private void publishDummyDataToTopic(final String inputTopic, final EmbeddedKafkaCluster cluster) {
+        final Properties props = new Properties();
+        props.put("acks", "all");
+        props.put("retries", 1);
+        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, cluster.bootstrapServers());
+        props.put(ProducerConfig.CLIENT_ID_CONFIG, "test-client");
+        props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
+        props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
+        final KafkaProducer<String, String> dummyProducer = new KafkaProducer<>(props);
+        dummyProducer.send(new ProducerRecord<String, String>(inputTopic, Integer.toString(4), Integer.toString(4)));

Review comment:
       It might be a good idea to send a slightly larger batch of data, for example I think in other integration tests we did like 10,000 records. We don't necessarily need that many here but Streams should be fast enough that we may as well do something like 1,000 - 5,000




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [kafka] ableegoldman commented on pull request #10554: Expand AdjustStreamThreadCountTest by writing some data to Kafka topics

[GitBox <gi...@apache.org>]

ableegoldman commented on pull request #10554:
URL: https://github.com/apache/kafka/pull/10554#issuecomment-827263009


   Thanks for the PR, I think this is step forward since just processing some data at all may have caught that cache bug a while back. But I wonder if we can take it a step further in this PR and also try to validate that some data did get processed in each of the tests?
   
   I can think of two ways of doing this. One is obviously just to wait on some data showing up in the output topic. Not sure what topology all of these tests are running at the moment but if there's a 1:1 ratio between input records and output then you can just wait until however many records you pipe in. There should be some methods in IntegrationTestUtils that do this already, check out `waitUntilMinRecordsReceived`
   Another way would be to use a custom processor/transformer that sets a flag once all the expected data has been processed. But tbh that first method is probably the best since it tests end-to-end including writing to an output topic, plus you can use an existing IntegrationTestUtils method so there's less custom code needed


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org